Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.058928815s FAIL (2025-09-08 01:36:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.058955074s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.787792221s FAIL (2025-09-08 01:35:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 5.78788509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.042210242s FAIL (2025-09-08 01:35:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.04224018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.120443254s FAIL (2025-09-08 01:35:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.120470827s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.0646065s FAIL (2025-09-08 01:34:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.064629262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.019380773s FAIL (2025-09-08 01:34:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.019404429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.747991522s FAIL (2025-09-08 01:34:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 5.748022421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.887964892s FAIL (2025-09-08 01:33:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.887990811s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.086039915s FAIL (2025-09-08 01:33:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.086068921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.943273254s FAIL (2025-09-08 01:33:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 5.943296098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.085729966s FAIL (2025-09-08 01:32:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.085750755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.294708954s FAIL (2025-09-08 01:32:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.294735954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.180864224s FAIL (2025-09-08 01:32:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.180894893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.095227025s FAIL (2025-09-08 01:32:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.095306858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.997648124s FAIL (2025-09-08 01:31:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 5.9976767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.01523636s FAIL (2025-09-08 01:31:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.015274183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.929579303s FAIL (2025-09-08 01:31:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.929613368s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.145560525s FAIL (2025-09-08 01:30:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.145588779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.166033919s FAIL (2025-09-08 01:30:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.166071771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.095680534s FAIL (2025-09-08 01:30:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.095702767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.050029438s FAIL (2025-09-08 01:29:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.050054755s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.132050931s FAIL (2025-09-08 01:28:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.132073623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.028476599s FAIL (2025-09-08 01:28:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.028514932s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.082853147s FAIL (2025-09-08 01:28:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.08289109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.090097329s FAIL (2025-09-08 01:27:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.090127195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.049258016s FAIL (2025-09-08 01:27:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.049283814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.125959665s FAIL (2025-09-08 01:27:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.125990924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.886549781s FAIL (2025-09-08 01:27:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 5.886573436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.780703456s FAIL (2025-09-08 01:26:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 5.780729225s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.137019121s FAIL (2025-09-08 01:26:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.13704501s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.023418655s FAIL (2025-09-08 01:26:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.023449173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.905845327s FAIL (2025-09-08 01:26:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.905863331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.80775449s FAIL (2025-09-08 01:25:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 5.807779928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.958757462s FAIL (2025-09-08 01:25:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.958806867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.218576657s FAIL (2025-09-08 01:24:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.218600233s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.03142621s FAIL (2025-09-08 01:24:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.031456237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.893855866s FAIL (2025-09-08 01:23:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.893898287s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.812173732s FAIL (2025-09-08 01:23:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 5.812212436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.974430949s FAIL (2025-09-08 01:23:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.974456587s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.101758224s FAIL (2025-09-08 01:22:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.101778993s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.874105777s FAIL (2025-09-08 01:22:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 5.874148499s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.114226331s FAIL (2025-09-08 01:22:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.114335941s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.030993744s FAIL (2025-09-08 01:21:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.031021087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.189102953s FAIL (2025-09-08 01:21:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.189127929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.996096247s FAIL (2025-09-08 01:21:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 5.996126546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.005217517s FAIL (2025-09-08 01:21:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.005259809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 13.520022635s FAIL (2025-09-08 01:20:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 13.520052763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 16.841479863s FAIL (2025-09-08 01:20:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 16.841536402s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.187534612s FAIL (2025-09-08 01:19:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 8 01:05:04.294020248 (35h36m40.333593368s) [upload=31295]: Starting upload Sep 8 01:05:04.294084921 (35h36m40.333658051s) [upload=31295]: Done starting upload Sep 8 01:05:04.294247722 (35h36m40.333820843s) [upload=31295]: Uploading segments... Sep 8 01:05:04.294382120 (35h36m40.33395524s) [upload=31295]: Got next segment. Inline: false Sep 8 01:05:04.294425233 (35h36m40.333998353s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:04.294430493 (35h36m40.334003613s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:04.866639830 (35h36m40.90621295s) [seg_pos={0 0}, upload=31295]: Begin upload segment... Sep 8 01:05:04.866689444 (35h36m40.906262564s) [seg_pos={0 0}, upload=31295]: Done begin upload segment. Sep 8 01:05:04.866714362 (35h36m40.906287482s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.866795087 (35h36m40.906368217s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867098647 (35h36m40.906671767s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867452745 (35h36m40.907025865s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867493333 (35h36m40.907066453s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867744964 (35h36m40.907318084s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.867895722 (35h36m40.907468842s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868025070 (35h36m40.9075982s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868298654 (35h36m40.907871784s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868314333 (35h36m40.907887453s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868345644 (35h36m40.907918764s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868574091 (35h36m40.908147211s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868676527 (35h36m40.908249647s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.868849467 (35h36m40.908422588s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869075991 (35h36m40.908649111s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869105667 (35h36m40.908678787s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869340898 (35h36m40.908914018s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:04.869452391 (35h36m40.909025511s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Uploading piece... Sep 8 01:05:06.395648000 (35h36m42.43522112s) [upload=31295]: Next returned nil segment Sep 8 01:05:06.395664983 (35h36m42.435238103s) [upload=31295]: Waiting for error group managing segments... Sep 8 01:05:06.790549857 (35h36m42.830122977s) [noise=true , address=145.79.227.4:4214 , piece_id=U5TRBOHRL2BAAM4A5S32YWJVF6XG2IHTKZKGAWTCB43B2SUMCCEQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.807695422 (35h36m42.847268542s) [noise=true , address=145.79.227.7:4303 , piece_id=AGSLJHU44JFJ26DRI4MSVKFHKCNCG5JPC656KQWUAACMFLDULWTA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.853444697 (35h36m42.893017817s) [noise=true , address=145.79.227.49:4339 , piece_id=CKUK5DZ4GMRYKR4SVVG4TMHUDRL26PB36WP6BE36MR52WDQ7FZCQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.866919158 (35h36m42.906492278s) [noise=true , address=145.79.227.6:4261 , piece_id=PCRBPC2FJY2SUU2PQI6D6C2B4UTO666YOIL7V4AGET4GS72OG6PA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.882171311 (35h36m42.921744431s) [noise=true , address=77.90.31.17:4785 , piece_id=TYR5SDV3BZMYIWYZMK2A6NGECSDFM2HUA3P3ZHXV6PI77M6ZD4HA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.900525018 (35h36m42.940098138s) [noise=true , address=145.79.227.47:4265 , piece_id=SD4Y2NSBSZTYA7WXK6NA7WB3KQFQDNR5KG5NIU4GT3SHHQEPMEXA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.904741876 (35h36m42.944314997s) [noise=true , address=145.79.227.38:4994 , piece_id=UNGTAICMB57HO5JEN76UBAU4OWUYDCTPKCNL53CTCXFYFDNP6CSQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.907456490 (35h36m42.94702961s) [noise=true , address=145.79.227.42:4121 , piece_id=WOQCCOI3KEXO6BX6M2NAA3GIOH7AKY7PIV6J5VKOLM4TKYG2M24Q, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.911368315 (35h36m42.950941445s) [noise=true , address=145.79.227.43:4183 , piece_id=TGUWLCJPLM4KNZWUIV6BN7GHSBI7RUITCFNWQJNDH4QWS5575VRQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.925378681 (35h36m42.964951841s) [noise=true , address=145.79.227.18:4031 , piece_id=5CBZ6G5GEZDAONDHCAQ2C43PSJQHLDMDQJXRDAOTFXQHUUAOIOPA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.928129575 (35h36m42.967702695s) [noise=true , address=145.79.227.37:4594 , piece_id=YBZ65ZVXT5TRTMUKUOPBSI5ED7XUDPFJMYZ2SN3GNA73NRFPE5ZQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.937432024 (35h36m42.977005144s) [noise=true , address=145.79.227.14:4845 , piece_id=2KKOOINYEZTZGBTTNV3NAL4LWJ7AUPBWOJHLFHXM4W5SSRIQ4QZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.938572887 (35h36m42.978146007s) [noise=true , address=145.79.227.33:4895 , piece_id=U5GXBZAURUDHR677GV6VHUVALA6TRHV4F6GSUO23AUE22WPALTJQ, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966722898 (35h36m43.006296018s) [noise=true , address=145.79.227.17:4975 , piece_id=J5TSDS2NM6Y35KKXIWOE4NRWDVKU3AQX2OVIIIFMP3AJXWDJJLZA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: <nil> Sep 8 01:05:06.966753977 (35h36m43.006327097s) [seg_pos={0 0}, upload=31295]: Segment reached optimal threshold of 14 pieces. Sep 8 01:05:06.966909785 (35h36m43.006482905s) [noise=true , address=145.79.227.15:4887 , piece_id=YO2KJYJZT7XJY7IUUS6RSTVBJ4HKD2LF63X7LVG5CNXOJDFKT6IA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.966975080 (35h36m43.0065482s) [noise=true , address=145.79.227.42:4138 , piece_id=IZROO6FDSA7EBWDNPGWCE2KQ2XEQYAWT5KF55PDFBM3UQWXKJG6A, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967018092 (35h36m43.006591212s) [noise=true , address=145.79.227.41:4120 , piece_id=KIWCAOPQJJTDBS5NWAJ3SIQIX6FIDN6AIURL6XKCCPX7TDKEXDLA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967063319 (35h36m43.006636439s) [noise=true , address=145.79.227.21:4477 , piece_id=J2JAPZ3IBJJJXTCHZDG57265GTAPDANTSPLMSV75RATXY754LHQA, seg_pos={0 0}, upload=31295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY): protocol: expected piece hash; context canceled; EOF Sep 8 01:05:06.967076985 (35h36m43.006650095s) [seg_pos={0 0}, upload=31295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 8 01:05:06.967162378 (35h36m43.006735499s) [upload=31295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 8 01:05:06.967168179 (35h36m43.006741299s) [upload=31295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 8 01:05:07.310679357 (35h36m43.350252477s) [upload=31295]: Done uploading segments... Sep 8 01:05:07.310700318 (35h36m43.350273428s) [upload=31295]: Upload finished. err: <nil> Duration: 6.187563819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.968131422s FAIL (2025-09-08 01:18:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 8 01:05:12.757110512 (35h36m48.796683632s) [upload=31297]: Starting upload Sep 8 01:05:12.757174896 (35h36m48.796748016s) [upload=31297]: Done starting upload Sep 8 01:05:12.757256542 (35h36m48.796829662s) [upload=31297]: Uploading segments... Sep 8 01:05:12.757358648 (35h36m48.796931768s) [upload=31297]: Got next segment. Inline: false Sep 8 01:05:12.757392763 (35h36m48.796965883s) [upload=31297]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 8 01:05:12.757397031 (35h36m48.796970151s) [upload=31297]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 8 01:05:13.092807249 (35h36m49.132380369s) [seg_pos={0 0}, upload=31297]: Begin upload segment... Sep 8 01:05:13.092850261 (35h36m49.132423381s) [seg_pos={0 0}, upload=31297]: Done begin upload segment. Sep 8 01:05:13.092877734 (35h36m49.132450854s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.092884446 (35h36m49.132457567s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093230309 (35h36m49.132803429s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093271006 (35h36m49.132844126s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093493782 (35h36m49.133066902s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.093777766 (35h36m49.133350886s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094080435 (35h36m49.133653555s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094352857 (35h36m49.133925977s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094365460 (35h36m49.13393858s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094578719 (35h36m49.134151839s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094790153 (35h36m49.134363273s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.094951651 (35h36m49.134524772s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095364361 (35h36m49.134937481s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.095717928 (35h36m49.135291058s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096012852 (35h36m49.135585972s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:13.096260797 (35h36m49.135833927s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Uploading piece... Sep 8 01:05:14.873078281 (35h36m50.912651401s) [upload=31297]: Next returned nil segment Sep 8 01:05:14.873094051 (35h36m50.912667162s) [upload=31297]: Waiting for error group managing segments... Sep 8 01:05:15.202698280 (35h36m51.2422714s) [noise=true , address=45.91.116.5:4297 , piece_id=NHLHOA5HF5JR5JB2V2LCS44QN25SEDKT3BYWU446IW524U6H65HQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.209094027 (35h36m51.248667147s) [noise=true , address=77.90.31.15:4699 , piece_id=MC4RD2DNSBTA6GS4ZXZS6VO2BWHV7UBUWPNGGZVWWQJAUHMADBPQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.212692733 (35h36m51.252265853s) [noise=true , address=45.91.116.14:4189 , piece_id=IJYRXOOZSTU3NU2UNFNXSJI22FY2T7ZDJXJWUGDITAJTGAXGP3JQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.248887306 (35h36m51.288460426s) [noise=true , address=45.91.116.12:4225 , piece_id=UPPD3VNT43L2IQHFPCXGUGFGV36IWC2S3KI4JCUSCYOWDRVCW6VQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.254327434 (35h36m51.293900554s) [noise=true , address=45.91.116.6:4973 , piece_id=XMG3Q3IOTLSSKYGOQO6HOVBFF2XJ5T7NTKCUWVQ43AXJULRMMXHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.255605851 (35h36m51.295178971s) [noise=true , address=77.90.31.18:4806 , piece_id=5J5XUUNTCKDP7NAEOY7S5Z7AKGNSIRPPVDJUHS3SNWXIUTYLO6WA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265299779 (35h36m51.304872899s) [noise=true , address=77.90.31.17:4772 , piece_id=M3YW2HBA7FDYO7L5GZV4REBIFWULTAW5NHAG46XZPGMRDQCXWHHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.265911729 (35h36m51.305484849s) [noise=true , address=77.90.31.14:4663 , piece_id=YJEC7FCENEJKH62OHDSKPFKMJS3WMJCG5IYEHLHOZASJUKM3G7SQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.276901737 (35h36m51.316474857s) [noise=true , address=45.91.116.9:4263 , piece_id=QTVQMP54Y6VNETR5TBQWQ5WOYLQ7O2VQPTRNGD74Q5Q4HDZFYPMA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:15.387535187 (35h36m51.427108317s) [noise=true , address=77.90.31.10:4517 , piece_id=Z3KQUV23D72RCUTWQNI23PPLZ3FX3AL5RA7SKWJSSGCQYUAT5KJQ, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:16.537381878 (35h36m52.576954998s) [noise=true , address=45.91.116.13:4153 , piece_id=VLRPMW2DLA55XJ5O63NSURH6HOPAREMCNJMQ5FPZY3FSLILBOFPA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:17.070063103 (35h36m53.109636223s) [noise=true , address=45.91.116.13:4155 , piece_id=6ZCCXZMDSLCLWFY2XTKLJAP4ZDTP6WTHGLGUSVPISLJXVUNPZDXA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: <nil> Sep 8 01:05:42.069703118 (35h37m18.109276238s) [noise=true , address=145.79.227.13:4804 , piece_id=5DGX3H5EAZBOWFDERDVBZ4HDONUIUZBQYQBVP4BMOI7G2ILAVDBA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 188.166.225.170:33168->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.069918490 (35h37m18.109491611s) [noise=true , address=145.79.227.11:4729 , piece_id=KVPLH34LRLZEBDZ5JUGE7EDEY2Z6N6PWCOADOLGIBMKCUEXQBUHA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070009705 (35h37m18.109582825s) [noise=true , address=145.79.227.7:4293 , piece_id=NMDQ53GRSUXWJDWE4BXZGSJISBSH2HN267IZNGMOUVMRF5QG6N5A, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: read tcp 188.166.225.170:50614->145.79.227.7:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070048449 (35h37m18.109621569s) [noise=true , address=145.79.227.6:4259 , piece_id=OIYFCH2NOOROE6BKFEI2YNM72U6ZJCNMFP3HV6RZTTSXS2Z5KLCA, seg_pos={0 0}, upload=31297]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: read tcp 188.166.225.170:49928->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 8 01:05:42.070099257 (35h37m18.109672377s) [seg_pos={0 0}, upload=31297]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 8 01:05:42.335470264 (35h37m18.375043394s) [upload=31297]: Done uploading segments... Sep 8 01:05:42.335501313 (35h37m18.375074433s) [upload=31297]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.968160258s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)