Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X.....X..XXXX] duration: 30.00277735s trace: https://jaeger.dev.storj.io/trace/74dc44306aa6aef3 FAIL (2025-12-28 04:31:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:30:34.669392828 (541h1m34.540475883s) [upload=474473]: Starting upload Dec 28 04:30:34.669471758 (541h1m34.540554793s) [upload=474473]: Done starting upload Dec 28 04:30:34.669573131 (541h1m34.540656186s) [upload=474473]: Uploading segments... Dec 28 04:30:34.669729460 (541h1m34.540812505s) [upload=474473]: Got next segment. Inline: false Dec 28 04:30:34.669796637 (541h1m34.540879673s) [upload=474473]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:30:34.669842615 (541h1m34.54092567s) [upload=474473]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:30:34.911287843 (541h1m34.782370879s) [seg_pos={0 0}, upload=474473]: Begin upload segment... Dec 28 04:30:34.911331627 (541h1m34.782414672s) [seg_pos={0 0}, upload=474473]: Done begin upload segment. Dec 28 04:30:34.911384608 (541h1m34.782467663s) [noise=true , address=45.91.116.5:4297 , piece_id=X5Z7USZWNCZJIOJVHZCZ642BQVF3XAD4T3XNGJLJ2DAYWMOV67RQ, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.911405217 (541h1m34.782488252s) [noise=true , address=145.79.227.31:4797 , piece_id=D3XWRWUDO5YGGQGX3IHSTRVWJX5FGFVXS4UYBHG64VZGFQ46JCDQ, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.912476568 (541h1m34.783559603s) [noise=true , address=145.79.227.12:4765 , piece_id=GSG2753Q4CGBF45XO7UVS5CDXH5L4TC6G2BSSMQQBHRYZLBLXWKQ, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.913215985 (541h1m34.78429902s) [noise=true , address=145.79.227.37:4566 , piece_id=CAGLLGOAWBY7JIM3PDWUTHSFVS3XWMZAFCR6CQ247T4Y6H7G3NEA, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.914803578 (541h1m34.785886643s) [noise=true , address=145.79.227.47:4267 , piece_id=WNLZSZ4R4T6ECDHT4ZWM54EEUUSNC7COKZBPAULQZHHZKAJZ2RNQ, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.917508017 (541h1m34.788591062s) [noise=true , address=145.79.227.43:4158 , piece_id=KX5PWYUV6PSEJFNNNVMP75X7CRCA37PJYEJW5C27VFBZN4NCEQRQ, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.922209930 (541h1m34.793292965s) [noise=true , address=45.91.116.4:4939 , piece_id=VKCB2EOXTQ4KKJJMR3RJ2IUZAM2DXT2BG5VGLRYWVTMLRNRGJXKA, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.922594252 (541h1m34.793677287s) [noise=true , address=145.79.227.5:4222 , piece_id=CENXNLXCVMZZX3FLYRNZ5RUZ5Y2RSPGZS6CM6J2GQ4B5LJKXNFMA, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.925689576 (541h1m34.796772611s) [noise=true , address=145.79.227.38:4977 , piece_id=MTIPUDFDSTOGJBBDLAHHQLRIR46JV3TP4RD4MCJGM7TQUCKLVYPQ, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.926378597 (541h1m34.797461632s) [noise=true , address=45.91.116.14:4189 , piece_id=CIQYARKQNQGG36TBCVXUZZLK5GYHUTMGIZHIX74OQRHNY2JEBOIA, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.927736603 (541h1m34.798819648s) [noise=true , address=145.79.227.41:4086 , piece_id=WCFDKA4P52DXTAG42U5WPKPRNUMMIU6G5KYGW7EQJFJGHD7UHN5A, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.929320129 (541h1m34.800403174s) [noise=true , address=45.91.116.12:4228 , piece_id=H3W73ZN57KTMZPQEYPU5SILTN4CORJH42JXORGNN65NFBKSMMQIA, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.931128102 (541h1m34.802211137s) [noise=true , address=145.79.227.36:4519 , piece_id=4PMJIFSYOMWZNNB5Z2DM5JDHI3AXVZ7UJ7T7TAYVCRAAEWEYDR7A, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.931230547 (541h1m34.802313592s) [noise=true , address=145.79.227.39:4013 , piece_id=G7ROVFUD2NBGQSVPNKF7IQQI3ZDZJX3A7AOG27GWLUVLESCEZWMQ, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.933333841 (541h1m34.804416876s) [noise=true , address=145.79.227.6:4258 , piece_id=SSK2NN75HZPGL3YZZGZXJNHDYBFPCCJYCYYFIJAKXXXYH72MC3NA, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:34.933530806 (541h1m34.804613841s) [noise=true , address=145.79.227.32:4836 , piece_id=ZFW23IZL4N4P4BSUZ3T6QIFDKB3WPQHRZOHATWGIURTFVZQNXCCQ, seg_pos={0 0}, upload=474473]: Uploading piece... Dec 28 04:30:36.628725983 (541h1m36.499809028s) [upload=474473]: Next returned nil segment Dec 28 04:30:36.628748665 (541h1m36.4998317s) [upload=474473]: Waiting for error group managing segments... Dec 28 04:30:36.913410041 (541h1m36.784493076s) [noise=true , address=145.79.227.12:4765 , piece_id=GSG2753Q4CGBF45XO7UVS5CDXH5L4TC6G2BSSMQQBHRYZLBLXWKQ, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:36.919621719 (541h1m36.790704754s) [noise=true , address=145.79.227.31:4797 , piece_id=D3XWRWUDO5YGGQGX3IHSTRVWJX5FGFVXS4UYBHG64VZGFQ46JCDQ, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:36.926170088 (541h1m36.797253123s) [noise=true , address=145.79.227.39:4013 , piece_id=G7ROVFUD2NBGQSVPNKF7IQQI3ZDZJX3A7AOG27GWLUVLESCEZWMQ, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:36.932416932 (541h1m36.803499977s) [noise=true , address=145.79.227.6:4258 , piece_id=SSK2NN75HZPGL3YZZGZXJNHDYBFPCCJYCYYFIJAKXXXYH72MC3NA, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:36.935755860 (541h1m36.806838895s) [noise=true , address=145.79.227.47:4267 , piece_id=WNLZSZ4R4T6ECDHT4ZWM54EEUUSNC7COKZBPAULQZHHZKAJZ2RNQ, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:36.936684588 (541h1m36.807767623s) [noise=true , address=145.79.227.36:4519 , piece_id=4PMJIFSYOMWZNNB5Z2DM5JDHI3AXVZ7UJ7T7TAYVCRAAEWEYDR7A, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:36.941881815 (541h1m36.81296485s) [noise=true , address=145.79.227.37:4566 , piece_id=CAGLLGOAWBY7JIM3PDWUTHSFVS3XWMZAFCR6CQ247T4Y6H7G3NEA, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:36.945676680 (541h1m36.816759735s) [noise=true , address=145.79.227.32:4836 , piece_id=ZFW23IZL4N4P4BSUZ3T6QIFDKB3WPQHRZOHATWGIURTFVZQNXCCQ, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:36.950188470 (541h1m36.821271515s) [noise=true , address=145.79.227.5:4222 , piece_id=CENXNLXCVMZZX3FLYRNZ5RUZ5Y2RSPGZS6CM6J2GQ4B5LJKXNFMA, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:36.963227718 (541h1m36.834310753s) [noise=true , address=145.79.227.41:4086 , piece_id=WCFDKA4P52DXTAG42U5WPKPRNUMMIU6G5KYGW7EQJFJGHD7UHN5A, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:36.965975881 (541h1m36.837058916s) [noise=true , address=145.79.227.38:4977 , piece_id=MTIPUDFDSTOGJBBDLAHHQLRIR46JV3TP4RD4MCJGM7TQUCKLVYPQ, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:37.141628826 (541h1m37.012711871s) [noise=true , address=45.91.116.12:4228 , piece_id=H3W73ZN57KTMZPQEYPU5SILTN4CORJH42JXORGNN65NFBKSMMQIA, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:37.277040090 (541h1m37.148123125s) [noise=true , address=45.91.116.5:4297 , piece_id=X5Z7USZWNCZJIOJVHZCZ642BQVF3XAD4T3XNGJLJ2DAYWMOV67RQ, seg_pos={0 0}, upload=474473]: Done uploading piece. err: <nil> Dec 28 04:30:37.277063814 (541h1m37.14814686s) [seg_pos={0 0}, upload=474473]: Segment reached optimal threshold of 13 pieces. Dec 28 04:30:37.277263275 (541h1m37.14834631s) [noise=true , address=45.91.116.4:4939 , piece_id=VKCB2EOXTQ4KKJJMR3RJ2IUZAM2DXT2BG5VGLRYWVTMLRNRGJXKA, seg_pos={0 0}, upload=474473]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:50324->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:50324->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50324->45.91.116.4:4939: use of closed network connection Dec 28 04:30:37.277271049 (541h1m37.148354084s) [noise=true , address=45.91.116.4:4939 , piece_id=VKCB2EOXTQ4KKJJMR3RJ2IUZAM2DXT2BG5VGLRYWVTMLRNRGJXKA, seg_pos={0 0}, upload=474473]: Long tail cancellation due to optimal threshold Dec 28 04:30:37.277281520 (541h1m37.148364565s) [noise=true , address=145.79.227.43:4158 , piece_id=KX5PWYUV6PSEJFNNNVMP75X7CRCA37PJYEJW5C27VFBZN4NCEQRQ, seg_pos={0 0}, upload=474473]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 28 04:30:37.277289344 (541h1m37.148372389s) [noise=true , address=145.79.227.43:4158 , piece_id=KX5PWYUV6PSEJFNNNVMP75X7CRCA37PJYEJW5C27VFBZN4NCEQRQ, seg_pos={0 0}, upload=474473]: Long tail cancellation due to optimal threshold Dec 28 04:30:37.277480608 (541h1m37.148563653s) [noise=true , address=45.91.116.14:4189 , piece_id=CIQYARKQNQGG36TBCVXUZZLK5GYHUTMGIZHIX74OQRHNY2JEBOIA, seg_pos={0 0}, upload=474473]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 04:30:37.277489646 (541h1m37.148572681s) [noise=true , address=45.91.116.14:4189 , piece_id=CIQYARKQNQGG36TBCVXUZZLK5GYHUTMGIZHIX74OQRHNY2JEBOIA, seg_pos={0 0}, upload=474473]: Long tail cancellation due to optimal threshold Dec 28 04:30:37.277511367 (541h1m37.148594412s) [seg_pos={0 0}, upload=474473]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:30:37.277604244 (541h1m37.148687279s) [upload=474473]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:30:37.277609524 (541h1m37.148692559s) [upload=474473]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:30:37.557951826 (541h1m37.429034861s) [upload=474473]: Done uploading segments... Dec 28 04:30:37.558003294 (541h1m37.429086329s) [upload=474473]: Upload finished. err: <nil> Duration: 30.002864234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X......X......................] duration: 41.664679526s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:30:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 41.6647081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X.....X..XXX] duration: 30.004867067s trace: https://jaeger.dev.storj.io/trace/221f3086bcc84464 FAIL (2025-12-28 04:30:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:29:46.936818296 (541h0m46.807901331s) [upload=474459]: Starting upload Dec 28 04:29:46.936906062 (541h0m46.807989107s) [upload=474459]: Done starting upload Dec 28 04:29:46.936998829 (541h0m46.808081864s) [upload=474459]: Uploading segments... Dec 28 04:29:46.937156300 (541h0m46.808239355s) [upload=474459]: Got next segment. Inline: false Dec 28 04:29:46.937200123 (541h0m46.808283168s) [upload=474459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:29:46.937206515 (541h0m46.80828955s) [upload=474459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:29:47.180861623 (541h0m47.051944658s) [seg_pos={0 0}, upload=474459]: Begin upload segment... Dec 28 04:29:47.180927158 (541h0m47.052010203s) [seg_pos={0 0}, upload=474459]: Done begin upload segment. Dec 28 04:29:47.180991891 (541h0m47.052074936s) [noise=true , address=77.90.31.14:4663 , piece_id=UTC7MYXEPTEJB5AZHEBGHNQAA7HBPM5YHI4EQZY5IQ4KWMARJ57A, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.181007861 (541h0m47.052090896s) [noise=true , address=145.79.227.13:4804 , piece_id=HBQKR3EBUHNUKF5FEYAWQ43ND7A55IKTBMKJZDDZQPL4R24QNELA, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.181489729 (541h0m47.052572764s) [noise=true , address=145.79.227.18:4006 , piece_id=ASEPTIO4FPFRYOS4VTEO2TP3TD5PR5J56EWOLFZP5FN3VZY2D5EA, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.181921851 (541h0m47.053004886s) [noise=true , address=45.91.116.10:4083 , piece_id=HXRQLRMJXMVEXO24FKGZZ2MU6KTPFVNRJFGYMRVYAIZSIEAC7C2A, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.182223666 (541h0m47.053306711s) [noise=true , address=145.79.227.46:4601 , piece_id=GOX4HCNE3GAZUBGXGM6EFKCCGUA3WZULU542YBCSAXOECCNUNWRA, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.182504140 (541h0m47.053587185s) [noise=true , address=45.91.116.4:4940 , piece_id=NJBTYCN52OYECCIVUL46V3TOMVPBSKV2V4AWJULSQV3SFCZAASPA, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.182865127 (541h0m47.053948172s) [noise=true , address=77.90.31.16:4736 , piece_id=LG2YHCT73YZWVLESOVNF3QIUJOJQIZ244DSPZQB2NV3UGL4C7YQQ, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.183182832 (541h0m47.054265867s) [noise=true , address=145.79.227.45:4229 , piece_id=TSUUGFYZK65DF3TMWBNGXIELRJV5U3Z3XX5QOW4PYYUQB5Z74XXQ, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.183431766 (541h0m47.054514811s) [noise=true , address=145.79.227.27:4691 , piece_id=EW4IHWU5RFE62BTCKMMFC5K7FNDNYNPAE3ABJ7QYD7F2D2FQ6D2Q, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.183748490 (541h0m47.054831525s) [noise=true , address=145.79.227.51:4411 , piece_id=XA7NSAOJTDESRVTIT6L5VV56XVVX6LVBCVG7LT6FL4NNQYKHMXDA, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.184055704 (541h0m47.055138739s) [noise=true , address=77.90.31.13:4629 , piece_id=FT4TKXNQ2UNXINNBHG2RCV4E2U3GWZGBZIMNQQHCYKYR2TFKY6OQ, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.184360715 (541h0m47.05544375s) [noise=true , address=45.91.116.14:4189 , piece_id=3G3YEG7R5PD7DAPEBUFLPVHYYLNTWYM2AYZP7OB6ERYSIA2EMB6Q, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.184702486 (541h0m47.055785521s) [noise=true , address=45.91.116.11:4120 , piece_id=UBMO55TJ3H5TXH4MAXFEX7GNA3HHE5GF6GOEPGGQTWARSPRQRXEA, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.185118138 (541h0m47.056201183s) [noise=true , address=145.79.227.41:4086 , piece_id=3C5GCBM2YAQ64YWJFE4AYBS3AVS6D7CRCIGGDTOO2P3XV7SWUXBA, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.185517978 (541h0m47.056601024s) [noise=true , address=145.79.227.2:4113 , piece_id=27UDADLSAL6FEVV5XXSKPWWK766BUPMLKLF2O3QB6UZIVWXEZC4Q, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:47.185954340 (541h0m47.057037385s) [noise=true , address=145.79.227.31:4797 , piece_id=Y4HTAUE2DF5VXKMUIICAW34YAVHMBI7ZONE25TVJ7HOOKI3EEUFA, seg_pos={0 0}, upload=474459]: Uploading piece... Dec 28 04:29:48.906731414 (541h0m48.777814449s) [upload=474459]: Next returned nil segment Dec 28 04:29:48.906750661 (541h0m48.777833707s) [upload=474459]: Waiting for error group managing segments... Dec 28 04:29:49.194041414 (541h0m49.065124459s) [noise=true , address=145.79.227.45:4229 , piece_id=TSUUGFYZK65DF3TMWBNGXIELRJV5U3Z3XX5QOW4PYYUQB5Z74XXQ, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.201818031 (541h0m49.072901066s) [noise=true , address=145.79.227.46:4601 , piece_id=GOX4HCNE3GAZUBGXGM6EFKCCGUA3WZULU542YBCSAXOECCNUNWRA, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.204716791 (541h0m49.075799836s) [noise=true , address=145.79.227.18:4006 , piece_id=ASEPTIO4FPFRYOS4VTEO2TP3TD5PR5J56EWOLFZP5FN3VZY2D5EA, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.211255040 (541h0m49.082338085s) [noise=true , address=145.79.227.13:4804 , piece_id=HBQKR3EBUHNUKF5FEYAWQ43ND7A55IKTBMKJZDDZQPL4R24QNELA, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.211843551 (541h0m49.082926586s) [noise=true , address=145.79.227.41:4086 , piece_id=3C5GCBM2YAQ64YWJFE4AYBS3AVS6D7CRCIGGDTOO2P3XV7SWUXBA, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.214512052 (541h0m49.085595087s) [noise=true , address=145.79.227.27:4691 , piece_id=EW4IHWU5RFE62BTCKMMFC5K7FNDNYNPAE3ABJ7QYD7F2D2FQ6D2Q, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.328544428 (541h0m49.199627463s) [noise=true , address=145.79.227.31:4797 , piece_id=Y4HTAUE2DF5VXKMUIICAW34YAVHMBI7ZONE25TVJ7HOOKI3EEUFA, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.328567212 (541h0m49.199650247s) [noise=true , address=145.79.227.51:4411 , piece_id=XA7NSAOJTDESRVTIT6L5VV56XVVX6LVBCVG7LT6FL4NNQYKHMXDA, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.360484694 (541h0m49.231567739s) [noise=true , address=77.90.31.14:4663 , piece_id=UTC7MYXEPTEJB5AZHEBGHNQAA7HBPM5YHI4EQZY5IQ4KWMARJ57A, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.389700062 (541h0m49.260783117s) [noise=true , address=145.79.227.2:4113 , piece_id=27UDADLSAL6FEVV5XXSKPWWK766BUPMLKLF2O3QB6UZIVWXEZC4Q, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.438164292 (541h0m49.309247337s) [noise=true , address=77.90.31.13:4629 , piece_id=FT4TKXNQ2UNXINNBHG2RCV4E2U3GWZGBZIMNQQHCYKYR2TFKY6OQ, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.484569942 (541h0m49.355652987s) [noise=true , address=77.90.31.16:4736 , piece_id=LG2YHCT73YZWVLESOVNF3QIUJOJQIZ244DSPZQB2NV3UGL4C7YQQ, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.499648973 (541h0m49.370732018s) [noise=true , address=45.91.116.14:4189 , piece_id=3G3YEG7R5PD7DAPEBUFLPVHYYLNTWYM2AYZP7OB6ERYSIA2EMB6Q, seg_pos={0 0}, upload=474459]: Done uploading piece. err: <nil> Dec 28 04:29:49.499673801 (541h0m49.370756846s) [seg_pos={0 0}, upload=474459]: Segment reached optimal threshold of 13 pieces. Dec 28 04:29:49.499884091 (541h0m49.370967126s) [noise=true , address=45.91.116.11:4120 , piece_id=UBMO55TJ3H5TXH4MAXFEX7GNA3HHE5GF6GOEPGGQTWARSPRQRXEA, seg_pos={0 0}, upload=474459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 28 04:29:49.499890563 (541h0m49.370973598s) [noise=true , address=45.91.116.11:4120 , piece_id=UBMO55TJ3H5TXH4MAXFEX7GNA3HHE5GF6GOEPGGQTWARSPRQRXEA, seg_pos={0 0}, upload=474459]: Long tail cancellation due to optimal threshold Dec 28 04:29:49.500096866 (541h0m49.371179901s) [noise=true , address=45.91.116.4:4940 , piece_id=NJBTYCN52OYECCIVUL46V3TOMVPBSKV2V4AWJULSQV3SFCZAASPA, seg_pos={0 0}, upload=474459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 04:29:49.500104851 (541h0m49.371187886s) [noise=true , address=45.91.116.4:4940 , piece_id=NJBTYCN52OYECCIVUL46V3TOMVPBSKV2V4AWJULSQV3SFCZAASPA, seg_pos={0 0}, upload=474459]: Long tail cancellation due to optimal threshold Dec 28 04:29:49.500116945 (541h0m49.37119998s) [noise=true , address=45.91.116.10:4083 , piece_id=HXRQLRMJXMVEXO24FKGZZ2MU6KTPFVNRJFGYMRVYAIZSIEAC7C2A, seg_pos={0 0}, upload=474459]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:51846->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:51846->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51846->45.91.116.10:4083: use of closed network connection Dec 28 04:29:49.500146842 (541h0m49.371229877s) [noise=true , address=45.91.116.10:4083 , piece_id=HXRQLRMJXMVEXO24FKGZZ2MU6KTPFVNRJFGYMRVYAIZSIEAC7C2A, seg_pos={0 0}, upload=474459]: Long tail cancellation due to optimal threshold Dec 28 04:29:49.500210212 (541h0m49.371293247s) [seg_pos={0 0}, upload=474459]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:29:49.500292700 (541h0m49.371375745s) [upload=474459]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:29:49.500297759 (541h0m49.371380794s) [upload=474459]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:29:49.793346746 (541h0m49.664429791s) [upload=474459]: Done uploading segments... Dec 28 04:29:49.793365442 (541h0m49.664448477s) [upload=474459]: Upload finished. err: <nil> Duration: 30.004896034s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X.X..X...XX.....X......XX...X.X....X.X...XXXXX....XX..X....] duration: 5.002252195s trace: https://jaeger.dev.storj.io/trace/66705a8d1fc22b47 FAIL (2025-12-28 04:30:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002279888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X......X.....................] duration: 47.416254821s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:29:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 47.416314545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X.....X..XX] duration: 30.00289668s trace: https://jaeger.dev.storj.io/trace/1a489151de652660 FAIL (2025-12-28 04:29:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: ID4LKG6U774PFKM6FR6BSXNBVRAF7ULCIJRP7I5JKYTOZSRLLAFQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: UDTCKJOHJUOVP55MVJLWFZJZJSK5L47V3TOTFO6LAPYG6OBTCQNA) context deadline exceeded ======LOGS====== Dec 28 04:29:11.226441033 (541h0m11.097524069s) [upload=474450]: Starting upload Dec 28 04:29:11.226519963 (541h0m11.097603019s) [upload=474450]: Done starting upload Dec 28 04:29:11.226742357 (541h0m11.097825392s) [upload=474450]: Uploading segments... Dec 28 04:29:11.226925666 (541h0m11.098008711s) [upload=474450]: Got next segment. Inline: false Dec 28 04:29:11.226976172 (541h0m11.098059207s) [upload=474450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:29:11.226981322 (541h0m11.098064367s) [upload=474450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:29:11.470020129 (541h0m11.341103164s) [seg_pos={0 0}, upload=474450]: Begin upload segment... Dec 28 04:29:11.470066968 (541h0m11.341150003s) [seg_pos={0 0}, upload=474450]: Done begin upload segment. Dec 28 04:29:11.470096775 (541h0m11.34117982s) [noise=true , address=45.91.116.10:4082 , piece_id=UDTCKJOHJUOVP55MVJLWFZJZJSK5L47V3TOTFO6LAPYG6OBTCQNA, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.470155086 (541h0m11.341238131s) [noise=true , address=145.79.227.42:4123 , piece_id=2T6G4ZBYKYY567EZJ2UL25PCYNJANVSPN5LAUFP5QQ5NQF6WUVNA, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.470849838 (541h0m11.341932883s) [noise=true , address=145.79.227.4:4187 , piece_id=5QLUNRZJTUWTBOU2HQZBDS27JGK6OSPDXUBHCS5BFH4DGAFGN4EA, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.471350111 (541h0m11.342433156s) [noise=true , address=145.79.227.19:4077 , piece_id=R2VCQ6G7WV5INSC64AWSDERX7QGS2KPPMVP5I65XUXI5LII3V2VQ, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.471822871 (541h0m11.342905916s) [noise=true , address=145.79.227.47:4266 , piece_id=YBNWHAC3P2X7TBYTPNYP647477EQV23FCWLDN6HBCPA25VHHNHMQ, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.472251438 (541h0m11.343334473s) [noise=true , address=145.79.227.10:4402 , piece_id=J6VFXOEGOZZ732C3EBZEF5ELT3CUIBJ4YX4M7KOPZPLZW2ZWHGPA, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.472631811 (541h0m11.343714856s) [noise=true , address=145.79.227.29:4943 , piece_id=4ETEXH3E42X3VYAPMBADSGQZX6YBDNV34ZSXOJLEGVXZCV62NKPA, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.472852963 (541h0m11.343936018s) [noise=true , address=145.79.227.3:4150 , piece_id=TDFC66JYGC2ULSUENCCDAJ2UV2YDD3JGABEKAOUD44KZ3FEJMSRA, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.473013017 (541h0m11.344096052s) [noise=true , address=145.79.227.30:4761 , piece_id=K6LILMBNVO6DQFUHVTFDDVTOBL566XU66LM7MEGBDWI65HNWIQMA, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.473483193 (541h0m11.344566238s) [noise=true , address=145.79.227.50:4373 , piece_id=DMB44VKHSN5WY7GVOQLF2K2T2GUENEDOONVSTIR4Y3XDR4P5GGUQ, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.473937548 (541h0m11.345020593s) [noise=true , address=45.91.116.11:4118 , piece_id=ID4LKG6U774PFKM6FR6BSXNBVRAF7ULCIJRP7I5JKYTOZSRLLAFQ, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.474018131 (541h0m11.345101166s) [noise=true , address=145.79.227.2:4115 , piece_id=JF6C7LM3GM277M73Y3LASZMKRQE5RK5KS424GAQPGSGCSKHQ7EGA, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.474312692 (541h0m11.345395737s) [noise=true , address=45.91.116.8:4048 , piece_id=3JO3XMKYJGPZN6N6Y4AJSVO3IFVMIMDP5WZOON4IRN7OYBZWEDIQ, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.474547328 (541h0m11.345630363s) [noise=true , address=145.79.227.31:4797 , piece_id=SDLPEDWRKW5P57OG5XXO57Z2CEAWJVUBSOBOIPD3END5AMXPDL6A, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.474751758 (541h0m11.345834793s) [noise=true , address=77.90.31.13:4626 , piece_id=S36YBVLMEAN4LMIB4RVDBOYZ2BCZRUH5TA5IZXO5AQ5KWBS2UHQA, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:11.475143564 (541h0m11.346226599s) [noise=true , address=145.79.227.41:4085 , piece_id=6HKFWKN26BR5LN7EB7S7B7B6HLHKAMRSU62GHDHJ6U6I5YZP5O5A, seg_pos={0 0}, upload=474450]: Uploading piece... Dec 28 04:29:13.184446048 (541h0m13.055529083s) [upload=474450]: Next returned nil segment Dec 28 04:29:13.184459333 (541h0m13.055542378s) [upload=474450]: Waiting for error group managing segments... Dec 28 04:29:13.467791278 (541h0m13.338874323s) [noise=true , address=145.79.227.2:4115 , piece_id=JF6C7LM3GM277M73Y3LASZMKRQE5RK5KS424GAQPGSGCSKHQ7EGA, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.481442421 (541h0m13.352525466s) [noise=true , address=145.79.227.42:4123 , piece_id=2T6G4ZBYKYY567EZJ2UL25PCYNJANVSPN5LAUFP5QQ5NQF6WUVNA, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.486635229 (541h0m13.357718274s) [noise=true , address=145.79.227.19:4077 , piece_id=R2VCQ6G7WV5INSC64AWSDERX7QGS2KPPMVP5I65XUXI5LII3V2VQ, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.498750546 (541h0m13.369833591s) [noise=true , address=145.79.227.41:4085 , piece_id=6HKFWKN26BR5LN7EB7S7B7B6HLHKAMRSU62GHDHJ6U6I5YZP5O5A, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.505968931 (541h0m13.377051966s) [noise=true , address=145.79.227.50:4373 , piece_id=DMB44VKHSN5WY7GVOQLF2K2T2GUENEDOONVSTIR4Y3XDR4P5GGUQ, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.507371592 (541h0m13.378454627s) [noise=true , address=145.79.227.31:4797 , piece_id=SDLPEDWRKW5P57OG5XXO57Z2CEAWJVUBSOBOIPD3END5AMXPDL6A, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.508768081 (541h0m13.379851126s) [noise=true , address=145.79.227.4:4187 , piece_id=5QLUNRZJTUWTBOU2HQZBDS27JGK6OSPDXUBHCS5BFH4DGAFGN4EA, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.509033426 (541h0m13.380116471s) [noise=true , address=145.79.227.10:4402 , piece_id=J6VFXOEGOZZ732C3EBZEF5ELT3CUIBJ4YX4M7KOPZPLZW2ZWHGPA, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.514004321 (541h0m13.385087366s) [noise=true , address=145.79.227.3:4150 , piece_id=TDFC66JYGC2ULSUENCCDAJ2UV2YDD3JGABEKAOUD44KZ3FEJMSRA, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.522300328 (541h0m13.393383373s) [noise=true , address=145.79.227.47:4266 , piece_id=YBNWHAC3P2X7TBYTPNYP647477EQV23FCWLDN6HBCPA25VHHNHMQ, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.839238531 (541h0m13.710321566s) [noise=true , address=45.91.116.11:4118 , piece_id=ID4LKG6U774PFKM6FR6BSXNBVRAF7ULCIJRP7I5JKYTOZSRLLAFQ, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.842124916 (541h0m13.713207952s) [noise=true , address=45.91.116.10:4082 , piece_id=UDTCKJOHJUOVP55MVJLWFZJZJSK5L47V3TOTFO6LAPYG6OBTCQNA, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.925619918 (541h0m13.796702963s) [noise=true , address=45.91.116.8:4048 , piece_id=3JO3XMKYJGPZN6N6Y4AJSVO3IFVMIMDP5WZOON4IRN7OYBZWEDIQ, seg_pos={0 0}, upload=474450]: Done uploading piece. err: <nil> Dec 28 04:29:13.925649443 (541h0m13.796732478s) [seg_pos={0 0}, upload=474450]: Segment reached optimal threshold of 13 pieces. Dec 28 04:29:13.925983029 (541h0m13.797066124s) [noise=true , address=145.79.227.30:4761 , piece_id=K6LILMBNVO6DQFUHVTFDDVTOBL566XU66LM7MEGBDWI65HNWIQMA, seg_pos={0 0}, upload=474450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 04:29:13.925993509 (541h0m13.797076544s) [noise=true , address=145.79.227.30:4761 , piece_id=K6LILMBNVO6DQFUHVTFDDVTOBL566XU66LM7MEGBDWI65HNWIQMA, seg_pos={0 0}, upload=474450]: Long tail cancellation due to optimal threshold Dec 28 04:29:13.926069002 (541h0m13.797152057s) [noise=true , address=145.79.227.29:4943 , piece_id=4ETEXH3E42X3VYAPMBADSGQZX6YBDNV34ZSXOJLEGVXZCV62NKPA, seg_pos={0 0}, upload=474450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 28 04:29:13.926088830 (541h0m13.797171875s) [noise=true , address=145.79.227.29:4943 , piece_id=4ETEXH3E42X3VYAPMBADSGQZX6YBDNV34ZSXOJLEGVXZCV62NKPA, seg_pos={0 0}, upload=474450]: Long tail cancellation due to optimal threshold Dec 28 04:29:13.926302387 (541h0m13.797385432s) [noise=true , address=77.90.31.13:4626 , piece_id=S36YBVLMEAN4LMIB4RVDBOYZ2BCZRUH5TA5IZXO5AQ5KWBS2UHQA, seg_pos={0 0}, upload=474450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:54232->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:54232->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54232->77.90.31.13:4626: use of closed network connection Dec 28 04:29:13.926309570 (541h0m13.797392615s) [noise=true , address=77.90.31.13:4626 , piece_id=S36YBVLMEAN4LMIB4RVDBOYZ2BCZRUH5TA5IZXO5AQ5KWBS2UHQA, seg_pos={0 0}, upload=474450]: Long tail cancellation due to optimal threshold Dec 28 04:29:13.926332815 (541h0m13.79741586s) [seg_pos={0 0}, upload=474450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:29:13.926434889 (541h0m13.797517944s) [upload=474450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:29:13.926440460 (541h0m13.797523505s) [upload=474450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:29:14.215152880 (541h0m14.086235925s) [upload=474450]: Done uploading segments... Dec 28 04:29:14.215179140 (541h0m14.086262175s) [upload=474450]: Upload finished. err: <nil> Duration: 30.002924582s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX....XX..X...] duration: 4.174073754s trace: https://jaeger.dev.storj.io/trace/66705a8d1fc22b47 FAIL (2025-12-28 04:29:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.174096928s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX....XX..X..] duration: 4.231595007s trace: https://jaeger.dev.storj.io/trace/66705a8d1fc22b47 FAIL (2025-12-28 04:29:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.231623911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X......X....................] duration: 1m0.001903412s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:29:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 1m0.001930315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X.....X..X] duration: 30.002341711s trace: https://jaeger.dev.storj.io/trace/46c40b11b910b85e FAIL (2025-12-28 04:28:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: RGO3BCZOXZLDTOJ6RLS3GV53SMTKHOAYZUGB65HNCX2LRTHVLPDQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: NQBWSEKS526VECBO2NA33SHT3RZ7ZAPBEPX7DZ3OFT3C325LTYPQ) context deadline exceeded ======LOGS====== 3728894 (540h59m6.934811939s) [upload=474434]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:28:07.063736188 (540h59m6.934819223s) [upload=474434]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:28:07.303285607 (540h59m7.174368652s) [seg_pos={0 0}, upload=474434]: Begin upload segment... Dec 28 04:28:07.303346333 (540h59m7.174429378s) [seg_pos={0 0}, upload=474434]: Done begin upload segment. Dec 28 04:28:07.303382892 (540h59m7.174465937s) [noise=true , address=145.79.227.41:4085 , piece_id=RYYWAI6GH6EX3NUH56GLDR4IBL4WRYWMHR6IH2VJT5IVBDMQ7KRA, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.303393512 (540h59m7.174476558s) [noise=true , address=145.79.227.39:4015 , piece_id=364PUJ2MSRT5DQE4BLO6G2W2BQCSOZWRFR2ESODH62Z42GX3ELPQ, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.303455551 (540h59m7.174538586s) [noise=true , address=77.90.31.14:4664 , piece_id=B66626SJXMYQSTFQP4YR73P44UJR2P5D6PIGHO6ULNFOUN4FXHSQ, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.304074349 (540h59m7.175157394s) [noise=true , address=145.79.227.18:4007 , piece_id=R6XYV6JG4MAIYIFHL5Y3IPWB4BYDFAPYUTEMA63ZMYGU4NRUBSBA, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.304618335 (540h59m7.17570138s) [noise=true , address=145.79.227.35:4482 , piece_id=TLHOR7O22NV5PNI6RH22ZFBGNINR6UYHHKTEGS2SU6MEZGISCYPA, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.305111735 (540h59m7.17619478s) [noise=true , address=145.79.227.6:4257 , piece_id=NKT34DDMOUIR7LBQDFWPHKHOEKOGU42EUMRXJBHZI3FJ2AGZEYIQ, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.305549017 (540h59m7.176632062s) [noise=true , address=77.90.31.16:4736 , piece_id=RGO3BCZOXZLDTOJ6RLS3GV53SMTKHOAYZUGB65HNCX2LRTHVLPDQ, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.306002240 (540h59m7.177085275s) [noise=true , address=45.91.116.6:4973 , piece_id=NQBWSEKS526VECBO2NA33SHT3RZ7ZAPBEPX7DZ3OFT3C325LTYPQ, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.306420246 (540h59m7.177503291s) [noise=true , address=145.79.227.14:4839 , piece_id=PQ3HAZY3Q2N4RBAUT7FUKXLTGXDLXNRMYWOJMEYEBIX7K2EGPBQQ, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.306977747 (540h59m7.178060782s) [noise=true , address=145.79.227.46:4603 , piece_id=SZ6DMKDLT7OURN3LKT3TRQ4VBMOKI3KJP73KTJKDAQ7YERF4HLPA, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.307464224 (540h59m7.178547259s) [noise=true , address=45.91.116.12:4227 , piece_id=ZGIZJ7GXX3UNCA6NGUKBEA7PWS2IEME3Z6MADB7SWQ2QYN5YH6NA, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.307944689 (540h59m7.179027724s) [noise=true , address=145.79.227.5:4223 , piece_id=GDA7HDCMVTL4JON2EIJVII4OJWC3TQMZ4HYB6MHJ6TWKJTIX2Q3A, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.309904430 (540h59m7.180987465s) [noise=true , address=145.79.227.13:4804 , piece_id=KRVBA5EMWYTIE6LMBW4FOO7VXOAV3EHP4DVBEKXR6OBCXFVJO62A, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.311634455 (540h59m7.18271749s) [noise=true , address=45.91.116.11:4119 , piece_id=VNQZBIRZBH5DBO5ZKLGPHJPU5TLQGCGD7YU4BMJYVWLH2HAXKXEQ, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.314859465 (540h59m7.18594251s) [noise=true , address=145.79.227.50:4373 , piece_id=43XUKDDLPAZ4LC6BNR3N4MXM5UX6D6JMHG4DPAHF4G75FO6PGTYA, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:07.316690461 (540h59m7.187773506s) [noise=true , address=45.91.116.13:4153 , piece_id=N4SNNA35YXBLNBNUKS57WQUGXYHIEBMEMJQWA7PFK4JKHIW6NWLA, seg_pos={0 0}, upload=474434]: Uploading piece... Dec 28 04:28:09.027700770 (540h59m8.898783816s) [upload=474434]: Next returned nil segment Dec 28 04:28:09.027721549 (540h59m8.898804594s) [upload=474434]: Waiting for error group managing segments... Dec 28 04:28:09.315087886 (540h59m9.186170922s) [noise=true , address=145.79.227.46:4603 , piece_id=SZ6DMKDLT7OURN3LKT3TRQ4VBMOKI3KJP73KTJKDAQ7YERF4HLPA, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.315326119 (540h59m9.186409154s) [noise=true , address=145.79.227.41:4085 , piece_id=RYYWAI6GH6EX3NUH56GLDR4IBL4WRYWMHR6IH2VJT5IVBDMQ7KRA, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.319441606 (540h59m9.190524641s) [noise=true , address=145.79.227.50:4373 , piece_id=43XUKDDLPAZ4LC6BNR3N4MXM5UX6D6JMHG4DPAHF4G75FO6PGTYA, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.324309706 (540h59m9.195392741s) [noise=true , address=145.79.227.39:4015 , piece_id=364PUJ2MSRT5DQE4BLO6G2W2BQCSOZWRFR2ESODH62Z42GX3ELPQ, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.326133859 (540h59m9.197216904s) [noise=true , address=145.79.227.14:4839 , piece_id=PQ3HAZY3Q2N4RBAUT7FUKXLTGXDLXNRMYWOJMEYEBIX7K2EGPBQQ, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.340038444 (540h59m9.211121479s) [noise=true , address=145.79.227.13:4804 , piece_id=KRVBA5EMWYTIE6LMBW4FOO7VXOAV3EHP4DVBEKXR6OBCXFVJO62A, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.340573492 (540h59m9.211656527s) [noise=true , address=145.79.227.5:4223 , piece_id=GDA7HDCMVTL4JON2EIJVII4OJWC3TQMZ4HYB6MHJ6TWKJTIX2Q3A, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.377575448 (540h59m9.248658483s) [noise=true , address=145.79.227.35:4482 , piece_id=TLHOR7O22NV5PNI6RH22ZFBGNINR6UYHHKTEGS2SU6MEZGISCYPA, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.401419939 (540h59m9.272502974s) [noise=true , address=145.79.227.6:4257 , piece_id=NKT34DDMOUIR7LBQDFWPHKHOEKOGU42EUMRXJBHZI3FJ2AGZEYIQ, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.409726445 (540h59m9.28080949s) [noise=true , address=145.79.227.18:4007 , piece_id=R6XYV6JG4MAIYIFHL5Y3IPWB4BYDFAPYUTEMA63ZMYGU4NRUBSBA, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.532806539 (540h59m9.403889604s) [noise=true , address=45.91.116.12:4227 , piece_id=ZGIZJ7GXX3UNCA6NGUKBEA7PWS2IEME3Z6MADB7SWQ2QYN5YH6NA, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.586005593 (540h59m9.457088628s) [noise=true , address=77.90.31.16:4736 , piece_id=RGO3BCZOXZLDTOJ6RLS3GV53SMTKHOAYZUGB65HNCX2LRTHVLPDQ, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.688849044 (540h59m9.559932079s) [noise=true , address=45.91.116.6:4973 , piece_id=NQBWSEKS526VECBO2NA33SHT3RZ7ZAPBEPX7DZ3OFT3C325LTYPQ, seg_pos={0 0}, upload=474434]: Done uploading piece. err: <nil> Dec 28 04:28:09.688869915 (540h59m9.55995296s) [seg_pos={0 0}, upload=474434]: Segment reached optimal threshold of 13 pieces. Dec 28 04:28:09.689050939 (540h59m9.560133984s) [noise=true , address=45.91.116.11:4119 , piece_id=VNQZBIRZBH5DBO5ZKLGPHJPU5TLQGCGD7YU4BMJYVWLH2HAXKXEQ, seg_pos={0 0}, upload=474434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:58846->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:58846->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->45.91.116.11:4119: use of closed network connection Dec 28 04:28:09.689057571 (540h59m9.560140607s) [noise=true , address=45.91.116.11:4119 , piece_id=VNQZBIRZBH5DBO5ZKLGPHJPU5TLQGCGD7YU4BMJYVWLH2HAXKXEQ, seg_pos={0 0}, upload=474434]: Long tail cancellation due to optimal threshold Dec 28 04:28:09.689182269 (540h59m9.560265304s) [noise=true , address=77.90.31.14:4664 , piece_id=B66626SJXMYQSTFQP4YR73P44UJR2P5D6PIGHO6ULNFOUN4FXHSQ, seg_pos={0 0}, upload=474434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:58182->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:58182->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58182->77.90.31.14:4664: use of closed network connection Dec 28 04:28:09.689210092 (540h59m9.560293128s) [noise=true , address=77.90.31.14:4664 , piece_id=B66626SJXMYQSTFQP4YR73P44UJR2P5D6PIGHO6ULNFOUN4FXHSQ, seg_pos={0 0}, upload=474434]: Long tail cancellation due to optimal threshold Dec 28 04:28:09.689399022 (540h59m9.560482057s) [noise=true , address=45.91.116.13:4153 , piece_id=N4SNNA35YXBLNBNUKS57WQUGXYHIEBMEMJQWA7PFK4JKHIW6NWLA, seg_pos={0 0}, upload=474434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:38634->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:38634->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: context canceled Dec 28 04:28:09.689405885 (540h59m9.56048892s) [noise=true , address=45.91.116.13:4153 , piece_id=N4SNNA35YXBLNBNUKS57WQUGXYHIEBMEMJQWA7PFK4JKHIW6NWLA, seg_pos={0 0}, upload=474434]: Long tail cancellation due to optimal threshold Dec 28 04:28:09.689423168 (540h59m9.560506203s) [seg_pos={0 0}, upload=474434]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:28:09.689671922 (540h59m9.560754957s) [upload=474434]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:28:09.689678574 (540h59m9.560761619s) [upload=474434]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:28:09.968309258 (540h59m9.839392303s) [upload=474434]: Done uploading segments... Dec 28 04:28:09.968327763 (540h59m9.839410798s) [upload=474434]: Upload finished. err: <nil> Duration: 30.002373843s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [..X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX....XX..X.] duration: 3.490316526s trace: https://jaeger.dev.storj.io/trace/66705a8d1fc22b47 FAIL (2025-12-28 04:28:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490520625s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X.....X..] duration: 23.738951324s trace: https://jaeger.dev.storj.io/trace/14907578b36495a8 FAIL (2025-12-28 04:27:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:25:59.560486486 (540h56m59.431569521s) [upload=474401]: Starting upload Dec 28 04:25:59.560536621 (540h56m59.431619666s) [upload=474401]: Done starting upload Dec 28 04:25:59.560631011 (540h56m59.431714046s) [upload=474401]: Uploading segments... Dec 28 04:25:59.560866370 (540h56m59.431949415s) [upload=474401]: Got next segment. Inline: false Dec 28 04:25:59.560920733 (540h56m59.432003768s) [upload=474401]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:25:59.560926544 (540h56m59.432009589s) [upload=474401]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:25:59.805908438 (540h56m59.676991473s) [seg_pos={0 0}, upload=474401]: Begin upload segment... Dec 28 04:25:59.805951910 (540h56m59.677034945s) [seg_pos={0 0}, upload=474401]: Done begin upload segment. Dec 28 04:25:59.805984472 (540h56m59.677067508s) [noise=true , address=145.79.227.46:4603 , piece_id=T3RERBY56YYGPWWDDD4Y5PY656WY7NSQ65V3RJD57YOXWXTPUCVA, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.805999010 (540h56m59.677082045s) [noise=true , address=45.91.116.12:4227 , piece_id=T44BWUJIVLV7HQGZD5U33NSULAOKKTZTNK2VYGHGLQEK4ZYKBQWQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.806516866 (540h56m59.677599911s) [noise=true , address=45.91.116.14:4192 , piece_id=BXCYYKBQDQMLXDGUGOPOFJ4PVUBNFNEUEY24WAIVU5XG6M6ZWKXQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.806994846 (540h56m59.678077881s) [noise=true , address=145.79.227.6:4258 , piece_id=BPJYJ63TARANBCABPTOUAY3LID4JOR2OU2K6CCJRXLBUJEZUJPZA, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.807621610 (540h56m59.678704645s) [noise=true , address=77.90.31.16:4735 , piece_id=TXRLNAYNL4C67B344IEQTMRY4LHFW6EPPY4C6R4WVZYNP7HO57IQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.808427624 (540h56m59.67951067s) [noise=true , address=145.79.227.12:4767 , piece_id=VIDF2S4WSPDA3DNGW5CMHI3GPJ5U63F7NHX72O37SXPTXF65GTCQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.809398924 (540h56m59.680481969s) [noise=true , address=45.91.116.6:4974 , piece_id=GDCAHOLOMPPXYGQIOYYZPA2GIUIGDTMQ4JDNA7CXOSSBBHELHKOA, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.810153390 (540h56m59.681236425s) [noise=true , address=145.79.227.17:4970 , piece_id=GUVRTIDAMM5WUB62FST4FST2VRVYI5AE4PKM6F3OBFFXOAZANFDQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.810483519 (540h56m59.681566554s) [noise=true , address=145.79.227.42:4121 , piece_id=S6QSQCVGERLYWIFGNLSL2B26YLJAHQWI3IKZCZONDNDH4KPQ5LYQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.810901335 (540h56m59.68198437s) [noise=true , address=145.79.227.51:4409 , piece_id=QWXHBRRZBDZFCVPU4LE2JK7SUZZPGNMWDEZFVLZPNZWKRLJUDKSA, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.811209341 (540h56m59.682292387s) [noise=true , address=145.79.227.31:4797 , piece_id=KIEOBVR5DWTVV4XGBPJWDMDKQWAHGUF4WJRCVNTGFFX64NUWN7GQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.811598061 (540h56m59.682681097s) [noise=true , address=77.90.31.13:4627 , piece_id=F24TN6FHNB4UUOA2ABQCLXXLHL5BXQFI4LK3DEUWPIFXRRDDWUPQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.811982934 (540h56m59.683065979s) [noise=true , address=145.79.227.3:4151 , piece_id=S4566TMMN5I335BPA2QNMIGDCPBM4A4CWQWL23SBAHG4UUL5LDFQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.812837761 (540h56m59.683920797s) [noise=true , address=45.91.116.8:4046 , piece_id=ACXGYM4VQIKO275HQS2QDPF5WF4XBGBPDV2SNWIRETVJWPYOWTLQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.813161018 (540h56m59.684244063s) [noise=true , address=77.90.31.18:4808 , piece_id=HPSTMVIWCUXYS2IA6LNQCF3UJ5JLAKLGLZMXNV4YRYBO3PKRSU6Q, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.813474153 (540h56m59.684557188s) [noise=true , address=145.79.227.45:4230 , piece_id=P6S2WJPOMVRYHD5Z572BFER7YZNGKWP4W6I7C45JEA4NVPDHZY7A, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:26:01.522210703 (540h57m1.393293748s) [upload=474401]: Next returned nil segment Dec 28 04:26:01.522227825 (540h57m1.39331087s) [upload=474401]: Waiting for error group managing segments... Dec 28 04:26:01.808263178 (540h57m1.679346223s) [noise=true , address=145.79.227.46:4603 , piece_id=T3RERBY56YYGPWWDDD4Y5PY656WY7NSQ65V3RJD57YOXWXTPUCVA, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.812426425 (540h57m1.68350947s) [noise=true , address=145.79.227.42:4121 , piece_id=S6QSQCVGERLYWIFGNLSL2B26YLJAHQWI3IKZCZONDNDH4KPQ5LYQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.817179385 (540h57m1.68826243s) [noise=true , address=145.79.227.51:4409 , piece_id=QWXHBRRZBDZFCVPU4LE2JK7SUZZPGNMWDEZFVLZPNZWKRLJUDKSA, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.830780021 (540h57m1.701863066s) [noise=true , address=145.79.227.45:4230 , piece_id=P6S2WJPOMVRYHD5Z572BFER7YZNGKWP4W6I7C45JEA4NVPDHZY7A, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.837436256 (540h57m1.708519301s) [noise=true , address=145.79.227.12:4767 , piece_id=VIDF2S4WSPDA3DNGW5CMHI3GPJ5U63F7NHX72O37SXPTXF65GTCQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.838357840 (540h57m1.709440885s) [noise=true , address=145.79.227.6:4258 , piece_id=BPJYJ63TARANBCABPTOUAY3LID4JOR2OU2K6CCJRXLBUJEZUJPZA, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.841884195 (540h57m1.71296724s) [noise=true , address=145.79.227.31:4797 , piece_id=KIEOBVR5DWTVV4XGBPJWDMDKQWAHGUF4WJRCVNTGFFX64NUWN7GQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.925327597 (540h57m1.796410692s) [noise=true , address=145.79.227.17:4970 , piece_id=GUVRTIDAMM5WUB62FST4FST2VRVYI5AE4PKM6F3OBFFXOAZANFDQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.135086373 (540h57m2.006169419s) [noise=true , address=145.79.227.3:4151 , piece_id=S4566TMMN5I335BPA2QNMIGDCPBM4A4CWQWL23SBAHG4UUL5LDFQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.193956286 (540h57m2.065039331s) [noise=true , address=77.90.31.16:4735 , piece_id=TXRLNAYNL4C67B344IEQTMRY4LHFW6EPPY4C6R4WVZYNP7HO57IQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.202830042 (540h57m2.073913087s) [noise=true , address=45.91.116.14:4192 , piece_id=BXCYYKBQDQMLXDGUGOPOFJ4PVUBNFNEUEY24WAIVU5XG6M6ZWKXQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.398921315 (540h57m2.27000436s) [noise=true , address=45.91.116.6:4974 , piece_id=GDCAHOLOMPPXYGQIOYYZPA2GIUIGDTMQ4JDNA7CXOSSBBHELHKOA, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.504913541 (540h57m2.375996586s) [noise=true , address=45.91.116.8:4046 , piece_id=ACXGYM4VQIKO275HQS2QDPF5WF4XBGBPDV2SNWIRETVJWPYOWTLQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.504935043 (540h57m2.376018088s) [seg_pos={0 0}, upload=474401]: Segment reached optimal threshold of 13 pieces. Dec 28 04:26:02.505271352 (540h57m2.376354387s) [noise=true , address=77.90.31.13:4627 , piece_id=F24TN6FHNB4UUOA2ABQCLXXLHL5BXQFI4LK3DEUWPIFXRRDDWUPQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 28 04:26:02.505279287 (540h57m2.376362332s) [noise=true , address=77.90.31.13:4627 , piece_id=F24TN6FHNB4UUOA2ABQCLXXLHL5BXQFI4LK3DEUWPIFXRRDDWUPQ, seg_pos={0 0}, upload=474401]: Long tail cancellation due to optimal threshold Dec 28 04:26:02.505637149 (540h57m2.376720195s) [noise=true , address=45.91.116.12:4227 , piece_id=T44BWUJIVLV7HQGZD5U33NSULAOKKTZTNK2VYGHGLQEK4ZYKBQWQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: read tcp 188.166.225.170:38624->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:26:02.505655504 (540h57m2.376738559s) [noise=true , address=45.91.116.12:4227 , piece_id=T44BWUJIVLV7HQGZD5U33NSULAOKKTZTNK2VYGHGLQEK4ZYKBQWQ, seg_pos={0 0}, upload=474401]: Long tail cancellation due to optimal threshold Dec 28 04:26:02.505669291 (540h57m2.376752336s) [noise=true , address=77.90.31.18:4808 , piece_id=HPSTMVIWCUXYS2IA6LNQCF3UJ5JLAKLGLZMXNV4YRYBO3PKRSU6Q, seg_pos={0 0}, upload=474401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 04:26:02.505725848 (540h57m2.376808893s) [noise=true , address=77.90.31.18:4808 , piece_id=HPSTMVIWCUXYS2IA6LNQCF3UJ5JLAKLGLZMXNV4YRYBO3PKRSU6Q, seg_pos={0 0}, upload=474401]: Long tail cancellation due to optimal threshold Dec 28 04:26:02.505765804 (540h57m2.376848859s) [seg_pos={0 0}, upload=474401]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:26:02.505933593 (540h57m2.377016649s) [upload=474401]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:26:02.505942911 (540h57m2.377025956s) [upload=474401]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:26:02.827472437 (540h57m2.698555472s) [upload=474401]: Done uploading segments... Dec 28 04:26:02.827497225 (540h57m2.69858027s) [upload=474401]: Upload finished. err: <nil> Duration: 23.738980639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X......X...................] duration: 23.42818487s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:27:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 23.428212322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X..X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX....XX..X] duration: 5.001122022s trace: https://jaeger.dev.storj.io/trace/66705a8d1fc22b47 FAIL (2025-12-28 04:27:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001145087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X.....X.] duration: 5.834437339s trace: https://jaeger.dev.storj.io/trace/14907578b36495a8 FAIL (2025-12-28 04:27:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:25:59.560486486 (540h56m59.431569521s) [upload=474401]: Starting upload Dec 28 04:25:59.560536621 (540h56m59.431619666s) [upload=474401]: Done starting upload Dec 28 04:25:59.560631011 (540h56m59.431714046s) [upload=474401]: Uploading segments... Dec 28 04:25:59.560866370 (540h56m59.431949415s) [upload=474401]: Got next segment. Inline: false Dec 28 04:25:59.560920733 (540h56m59.432003768s) [upload=474401]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:25:59.560926544 (540h56m59.432009589s) [upload=474401]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:25:59.805908438 (540h56m59.676991473s) [seg_pos={0 0}, upload=474401]: Begin upload segment... Dec 28 04:25:59.805951910 (540h56m59.677034945s) [seg_pos={0 0}, upload=474401]: Done begin upload segment. Dec 28 04:25:59.805984472 (540h56m59.677067508s) [noise=true , address=145.79.227.46:4603 , piece_id=T3RERBY56YYGPWWDDD4Y5PY656WY7NSQ65V3RJD57YOXWXTPUCVA, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.805999010 (540h56m59.677082045s) [noise=true , address=45.91.116.12:4227 , piece_id=T44BWUJIVLV7HQGZD5U33NSULAOKKTZTNK2VYGHGLQEK4ZYKBQWQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.806516866 (540h56m59.677599911s) [noise=true , address=45.91.116.14:4192 , piece_id=BXCYYKBQDQMLXDGUGOPOFJ4PVUBNFNEUEY24WAIVU5XG6M6ZWKXQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.806994846 (540h56m59.678077881s) [noise=true , address=145.79.227.6:4258 , piece_id=BPJYJ63TARANBCABPTOUAY3LID4JOR2OU2K6CCJRXLBUJEZUJPZA, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.807621610 (540h56m59.678704645s) [noise=true , address=77.90.31.16:4735 , piece_id=TXRLNAYNL4C67B344IEQTMRY4LHFW6EPPY4C6R4WVZYNP7HO57IQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.808427624 (540h56m59.67951067s) [noise=true , address=145.79.227.12:4767 , piece_id=VIDF2S4WSPDA3DNGW5CMHI3GPJ5U63F7NHX72O37SXPTXF65GTCQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.809398924 (540h56m59.680481969s) [noise=true , address=45.91.116.6:4974 , piece_id=GDCAHOLOMPPXYGQIOYYZPA2GIUIGDTMQ4JDNA7CXOSSBBHELHKOA, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.810153390 (540h56m59.681236425s) [noise=true , address=145.79.227.17:4970 , piece_id=GUVRTIDAMM5WUB62FST4FST2VRVYI5AE4PKM6F3OBFFXOAZANFDQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.810483519 (540h56m59.681566554s) [noise=true , address=145.79.227.42:4121 , piece_id=S6QSQCVGERLYWIFGNLSL2B26YLJAHQWI3IKZCZONDNDH4KPQ5LYQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.810901335 (540h56m59.68198437s) [noise=true , address=145.79.227.51:4409 , piece_id=QWXHBRRZBDZFCVPU4LE2JK7SUZZPGNMWDEZFVLZPNZWKRLJUDKSA, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.811209341 (540h56m59.682292387s) [noise=true , address=145.79.227.31:4797 , piece_id=KIEOBVR5DWTVV4XGBPJWDMDKQWAHGUF4WJRCVNTGFFX64NUWN7GQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.811598061 (540h56m59.682681097s) [noise=true , address=77.90.31.13:4627 , piece_id=F24TN6FHNB4UUOA2ABQCLXXLHL5BXQFI4LK3DEUWPIFXRRDDWUPQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.811982934 (540h56m59.683065979s) [noise=true , address=145.79.227.3:4151 , piece_id=S4566TMMN5I335BPA2QNMIGDCPBM4A4CWQWL23SBAHG4UUL5LDFQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.812837761 (540h56m59.683920797s) [noise=true , address=45.91.116.8:4046 , piece_id=ACXGYM4VQIKO275HQS2QDPF5WF4XBGBPDV2SNWIRETVJWPYOWTLQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.813161018 (540h56m59.684244063s) [noise=true , address=77.90.31.18:4808 , piece_id=HPSTMVIWCUXYS2IA6LNQCF3UJ5JLAKLGLZMXNV4YRYBO3PKRSU6Q, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.813474153 (540h56m59.684557188s) [noise=true , address=145.79.227.45:4230 , piece_id=P6S2WJPOMVRYHD5Z572BFER7YZNGKWP4W6I7C45JEA4NVPDHZY7A, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:26:01.522210703 (540h57m1.393293748s) [upload=474401]: Next returned nil segment Dec 28 04:26:01.522227825 (540h57m1.39331087s) [upload=474401]: Waiting for error group managing segments... Dec 28 04:26:01.808263178 (540h57m1.679346223s) [noise=true , address=145.79.227.46:4603 , piece_id=T3RERBY56YYGPWWDDD4Y5PY656WY7NSQ65V3RJD57YOXWXTPUCVA, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.812426425 (540h57m1.68350947s) [noise=true , address=145.79.227.42:4121 , piece_id=S6QSQCVGERLYWIFGNLSL2B26YLJAHQWI3IKZCZONDNDH4KPQ5LYQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.817179385 (540h57m1.68826243s) [noise=true , address=145.79.227.51:4409 , piece_id=QWXHBRRZBDZFCVPU4LE2JK7SUZZPGNMWDEZFVLZPNZWKRLJUDKSA, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.830780021 (540h57m1.701863066s) [noise=true , address=145.79.227.45:4230 , piece_id=P6S2WJPOMVRYHD5Z572BFER7YZNGKWP4W6I7C45JEA4NVPDHZY7A, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.837436256 (540h57m1.708519301s) [noise=true , address=145.79.227.12:4767 , piece_id=VIDF2S4WSPDA3DNGW5CMHI3GPJ5U63F7NHX72O37SXPTXF65GTCQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.838357840 (540h57m1.709440885s) [noise=true , address=145.79.227.6:4258 , piece_id=BPJYJ63TARANBCABPTOUAY3LID4JOR2OU2K6CCJRXLBUJEZUJPZA, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.841884195 (540h57m1.71296724s) [noise=true , address=145.79.227.31:4797 , piece_id=KIEOBVR5DWTVV4XGBPJWDMDKQWAHGUF4WJRCVNTGFFX64NUWN7GQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.925327597 (540h57m1.796410692s) [noise=true , address=145.79.227.17:4970 , piece_id=GUVRTIDAMM5WUB62FST4FST2VRVYI5AE4PKM6F3OBFFXOAZANFDQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.135086373 (540h57m2.006169419s) [noise=true , address=145.79.227.3:4151 , piece_id=S4566TMMN5I335BPA2QNMIGDCPBM4A4CWQWL23SBAHG4UUL5LDFQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.193956286 (540h57m2.065039331s) [noise=true , address=77.90.31.16:4735 , piece_id=TXRLNAYNL4C67B344IEQTMRY4LHFW6EPPY4C6R4WVZYNP7HO57IQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.202830042 (540h57m2.073913087s) [noise=true , address=45.91.116.14:4192 , piece_id=BXCYYKBQDQMLXDGUGOPOFJ4PVUBNFNEUEY24WAIVU5XG6M6ZWKXQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.398921315 (540h57m2.27000436s) [noise=true , address=45.91.116.6:4974 , piece_id=GDCAHOLOMPPXYGQIOYYZPA2GIUIGDTMQ4JDNA7CXOSSBBHELHKOA, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.504913541 (540h57m2.375996586s) [noise=true , address=45.91.116.8:4046 , piece_id=ACXGYM4VQIKO275HQS2QDPF5WF4XBGBPDV2SNWIRETVJWPYOWTLQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.504935043 (540h57m2.376018088s) [seg_pos={0 0}, upload=474401]: Segment reached optimal threshold of 13 pieces. Dec 28 04:26:02.505271352 (540h57m2.376354387s) [noise=true , address=77.90.31.13:4627 , piece_id=F24TN6FHNB4UUOA2ABQCLXXLHL5BXQFI4LK3DEUWPIFXRRDDWUPQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 28 04:26:02.505279287 (540h57m2.376362332s) [noise=true , address=77.90.31.13:4627 , piece_id=F24TN6FHNB4UUOA2ABQCLXXLHL5BXQFI4LK3DEUWPIFXRRDDWUPQ, seg_pos={0 0}, upload=474401]: Long tail cancellation due to optimal threshold Dec 28 04:26:02.505637149 (540h57m2.376720195s) [noise=true , address=45.91.116.12:4227 , piece_id=T44BWUJIVLV7HQGZD5U33NSULAOKKTZTNK2VYGHGLQEK4ZYKBQWQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: read tcp 188.166.225.170:38624->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:26:02.505655504 (540h57m2.376738559s) [noise=true , address=45.91.116.12:4227 , piece_id=T44BWUJIVLV7HQGZD5U33NSULAOKKTZTNK2VYGHGLQEK4ZYKBQWQ, seg_pos={0 0}, upload=474401]: Long tail cancellation due to optimal threshold Dec 28 04:26:02.505669291 (540h57m2.376752336s) [noise=true , address=77.90.31.18:4808 , piece_id=HPSTMVIWCUXYS2IA6LNQCF3UJ5JLAKLGLZMXNV4YRYBO3PKRSU6Q, seg_pos={0 0}, upload=474401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 04:26:02.505725848 (540h57m2.376808893s) [noise=true , address=77.90.31.18:4808 , piece_id=HPSTMVIWCUXYS2IA6LNQCF3UJ5JLAKLGLZMXNV4YRYBO3PKRSU6Q, seg_pos={0 0}, upload=474401]: Long tail cancellation due to optimal threshold Dec 28 04:26:02.505765804 (540h57m2.376848859s) [seg_pos={0 0}, upload=474401]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:26:02.505933593 (540h57m2.377016649s) [upload=474401]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:26:02.505942911 (540h57m2.377025956s) [upload=474401]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:26:02.827472437 (540h57m2.698555472s) [upload=474401]: Done uploading segments... Dec 28 04:26:02.827497225 (540h57m2.69858027s) [upload=474401]: Upload finished. err: <nil> Duration: 5.834471463s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X..X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX....XX..] duration: 3.545755951s trace: https://jaeger.dev.storj.io/trace/2a1a75884003581b FAIL (2025-12-28 04:26:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.545783252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X......X..................] duration: 34.740209437s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:26:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 34.740225989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X.....X] duration: 30.004202169s trace: https://jaeger.dev.storj.io/trace/14907578b36495a8 FAIL (2025-12-28 04:26:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:25:59.560486486 (540h56m59.431569521s) [upload=474401]: Starting upload Dec 28 04:25:59.560536621 (540h56m59.431619666s) [upload=474401]: Done starting upload Dec 28 04:25:59.560631011 (540h56m59.431714046s) [upload=474401]: Uploading segments... Dec 28 04:25:59.560866370 (540h56m59.431949415s) [upload=474401]: Got next segment. Inline: false Dec 28 04:25:59.560920733 (540h56m59.432003768s) [upload=474401]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:25:59.560926544 (540h56m59.432009589s) [upload=474401]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:25:59.805908438 (540h56m59.676991473s) [seg_pos={0 0}, upload=474401]: Begin upload segment... Dec 28 04:25:59.805951910 (540h56m59.677034945s) [seg_pos={0 0}, upload=474401]: Done begin upload segment. Dec 28 04:25:59.805984472 (540h56m59.677067508s) [noise=true , address=145.79.227.46:4603 , piece_id=T3RERBY56YYGPWWDDD4Y5PY656WY7NSQ65V3RJD57YOXWXTPUCVA, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.805999010 (540h56m59.677082045s) [noise=true , address=45.91.116.12:4227 , piece_id=T44BWUJIVLV7HQGZD5U33NSULAOKKTZTNK2VYGHGLQEK4ZYKBQWQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.806516866 (540h56m59.677599911s) [noise=true , address=45.91.116.14:4192 , piece_id=BXCYYKBQDQMLXDGUGOPOFJ4PVUBNFNEUEY24WAIVU5XG6M6ZWKXQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.806994846 (540h56m59.678077881s) [noise=true , address=145.79.227.6:4258 , piece_id=BPJYJ63TARANBCABPTOUAY3LID4JOR2OU2K6CCJRXLBUJEZUJPZA, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.807621610 (540h56m59.678704645s) [noise=true , address=77.90.31.16:4735 , piece_id=TXRLNAYNL4C67B344IEQTMRY4LHFW6EPPY4C6R4WVZYNP7HO57IQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.808427624 (540h56m59.67951067s) [noise=true , address=145.79.227.12:4767 , piece_id=VIDF2S4WSPDA3DNGW5CMHI3GPJ5U63F7NHX72O37SXPTXF65GTCQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.809398924 (540h56m59.680481969s) [noise=true , address=45.91.116.6:4974 , piece_id=GDCAHOLOMPPXYGQIOYYZPA2GIUIGDTMQ4JDNA7CXOSSBBHELHKOA, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.810153390 (540h56m59.681236425s) [noise=true , address=145.79.227.17:4970 , piece_id=GUVRTIDAMM5WUB62FST4FST2VRVYI5AE4PKM6F3OBFFXOAZANFDQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.810483519 (540h56m59.681566554s) [noise=true , address=145.79.227.42:4121 , piece_id=S6QSQCVGERLYWIFGNLSL2B26YLJAHQWI3IKZCZONDNDH4KPQ5LYQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.810901335 (540h56m59.68198437s) [noise=true , address=145.79.227.51:4409 , piece_id=QWXHBRRZBDZFCVPU4LE2JK7SUZZPGNMWDEZFVLZPNZWKRLJUDKSA, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.811209341 (540h56m59.682292387s) [noise=true , address=145.79.227.31:4797 , piece_id=KIEOBVR5DWTVV4XGBPJWDMDKQWAHGUF4WJRCVNTGFFX64NUWN7GQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.811598061 (540h56m59.682681097s) [noise=true , address=77.90.31.13:4627 , piece_id=F24TN6FHNB4UUOA2ABQCLXXLHL5BXQFI4LK3DEUWPIFXRRDDWUPQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.811982934 (540h56m59.683065979s) [noise=true , address=145.79.227.3:4151 , piece_id=S4566TMMN5I335BPA2QNMIGDCPBM4A4CWQWL23SBAHG4UUL5LDFQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.812837761 (540h56m59.683920797s) [noise=true , address=45.91.116.8:4046 , piece_id=ACXGYM4VQIKO275HQS2QDPF5WF4XBGBPDV2SNWIRETVJWPYOWTLQ, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.813161018 (540h56m59.684244063s) [noise=true , address=77.90.31.18:4808 , piece_id=HPSTMVIWCUXYS2IA6LNQCF3UJ5JLAKLGLZMXNV4YRYBO3PKRSU6Q, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:25:59.813474153 (540h56m59.684557188s) [noise=true , address=145.79.227.45:4230 , piece_id=P6S2WJPOMVRYHD5Z572BFER7YZNGKWP4W6I7C45JEA4NVPDHZY7A, seg_pos={0 0}, upload=474401]: Uploading piece... Dec 28 04:26:01.522210703 (540h57m1.393293748s) [upload=474401]: Next returned nil segment Dec 28 04:26:01.522227825 (540h57m1.39331087s) [upload=474401]: Waiting for error group managing segments... Dec 28 04:26:01.808263178 (540h57m1.679346223s) [noise=true , address=145.79.227.46:4603 , piece_id=T3RERBY56YYGPWWDDD4Y5PY656WY7NSQ65V3RJD57YOXWXTPUCVA, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.812426425 (540h57m1.68350947s) [noise=true , address=145.79.227.42:4121 , piece_id=S6QSQCVGERLYWIFGNLSL2B26YLJAHQWI3IKZCZONDNDH4KPQ5LYQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.817179385 (540h57m1.68826243s) [noise=true , address=145.79.227.51:4409 , piece_id=QWXHBRRZBDZFCVPU4LE2JK7SUZZPGNMWDEZFVLZPNZWKRLJUDKSA, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.830780021 (540h57m1.701863066s) [noise=true , address=145.79.227.45:4230 , piece_id=P6S2WJPOMVRYHD5Z572BFER7YZNGKWP4W6I7C45JEA4NVPDHZY7A, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.837436256 (540h57m1.708519301s) [noise=true , address=145.79.227.12:4767 , piece_id=VIDF2S4WSPDA3DNGW5CMHI3GPJ5U63F7NHX72O37SXPTXF65GTCQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.838357840 (540h57m1.709440885s) [noise=true , address=145.79.227.6:4258 , piece_id=BPJYJ63TARANBCABPTOUAY3LID4JOR2OU2K6CCJRXLBUJEZUJPZA, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.841884195 (540h57m1.71296724s) [noise=true , address=145.79.227.31:4797 , piece_id=KIEOBVR5DWTVV4XGBPJWDMDKQWAHGUF4WJRCVNTGFFX64NUWN7GQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:01.925327597 (540h57m1.796410692s) [noise=true , address=145.79.227.17:4970 , piece_id=GUVRTIDAMM5WUB62FST4FST2VRVYI5AE4PKM6F3OBFFXOAZANFDQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.135086373 (540h57m2.006169419s) [noise=true , address=145.79.227.3:4151 , piece_id=S4566TMMN5I335BPA2QNMIGDCPBM4A4CWQWL23SBAHG4UUL5LDFQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.193956286 (540h57m2.065039331s) [noise=true , address=77.90.31.16:4735 , piece_id=TXRLNAYNL4C67B344IEQTMRY4LHFW6EPPY4C6R4WVZYNP7HO57IQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.202830042 (540h57m2.073913087s) [noise=true , address=45.91.116.14:4192 , piece_id=BXCYYKBQDQMLXDGUGOPOFJ4PVUBNFNEUEY24WAIVU5XG6M6ZWKXQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.398921315 (540h57m2.27000436s) [noise=true , address=45.91.116.6:4974 , piece_id=GDCAHOLOMPPXYGQIOYYZPA2GIUIGDTMQ4JDNA7CXOSSBBHELHKOA, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.504913541 (540h57m2.375996586s) [noise=true , address=45.91.116.8:4046 , piece_id=ACXGYM4VQIKO275HQS2QDPF5WF4XBGBPDV2SNWIRETVJWPYOWTLQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: <nil> Dec 28 04:26:02.504935043 (540h57m2.376018088s) [seg_pos={0 0}, upload=474401]: Segment reached optimal threshold of 13 pieces. Dec 28 04:26:02.505271352 (540h57m2.376354387s) [noise=true , address=77.90.31.13:4627 , piece_id=F24TN6FHNB4UUOA2ABQCLXXLHL5BXQFI4LK3DEUWPIFXRRDDWUPQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 28 04:26:02.505279287 (540h57m2.376362332s) [noise=true , address=77.90.31.13:4627 , piece_id=F24TN6FHNB4UUOA2ABQCLXXLHL5BXQFI4LK3DEUWPIFXRRDDWUPQ, seg_pos={0 0}, upload=474401]: Long tail cancellation due to optimal threshold Dec 28 04:26:02.505637149 (540h57m2.376720195s) [noise=true , address=45.91.116.12:4227 , piece_id=T44BWUJIVLV7HQGZD5U33NSULAOKKTZTNK2VYGHGLQEK4ZYKBQWQ, seg_pos={0 0}, upload=474401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: read tcp 188.166.225.170:38624->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:26:02.505655504 (540h57m2.376738559s) [noise=true , address=45.91.116.12:4227 , piece_id=T44BWUJIVLV7HQGZD5U33NSULAOKKTZTNK2VYGHGLQEK4ZYKBQWQ, seg_pos={0 0}, upload=474401]: Long tail cancellation due to optimal threshold Dec 28 04:26:02.505669291 (540h57m2.376752336s) [noise=true , address=77.90.31.18:4808 , piece_id=HPSTMVIWCUXYS2IA6LNQCF3UJ5JLAKLGLZMXNV4YRYBO3PKRSU6Q, seg_pos={0 0}, upload=474401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 04:26:02.505725848 (540h57m2.376808893s) [noise=true , address=77.90.31.18:4808 , piece_id=HPSTMVIWCUXYS2IA6LNQCF3UJ5JLAKLGLZMXNV4YRYBO3PKRSU6Q, seg_pos={0 0}, upload=474401]: Long tail cancellation due to optimal threshold Dec 28 04:26:02.505765804 (540h57m2.376848859s) [seg_pos={0 0}, upload=474401]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:26:02.505933593 (540h57m2.377016649s) [upload=474401]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:26:02.505942911 (540h57m2.377025956s) [upload=474401]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:26:02.827472437 (540h57m2.698555472s) [upload=474401]: Done uploading segments... Dec 28 04:26:02.827497225 (540h57m2.69858027s) [upload=474401]: Upload finished. err: <nil> Duration: 30.004226265s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X......X.................] duration: 40.864832085s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:25:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 40.864860137s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [..X..X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX....XX.] duration: 3.386360339s trace: https://jaeger.dev.storj.io/trace/2a1a75884003581b FAIL (2025-12-28 04:25:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.386385088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X.....] duration: 6.253915134s trace: https://jaeger.dev.storj.io/trace/52163586ab14fc00 FAIL (2025-12-28 04:25:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:21:16.187852237 (540h52m16.058935273s) [upload=474335]: Starting upload Dec 28 04:21:16.187919866 (540h52m16.059002901s) [upload=474335]: Done starting upload Dec 28 04:21:16.188001722 (540h52m16.059084767s) [upload=474335]: Uploading segments... Dec 28 04:21:16.188077466 (540h52m16.059160501s) [upload=474335]: Got next segment. Inline: false Dec 28 04:21:16.188113464 (540h52m16.05919658s) [upload=474335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:21:16.188121280 (540h52m16.059204325s) [upload=474335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:21:16.437024901 (540h52m16.308107956s) [seg_pos={0 0}, upload=474335]: Begin upload segment... Dec 28 04:21:16.437087139 (540h52m16.308170184s) [seg_pos={0 0}, upload=474335]: Done begin upload segment. Dec 28 04:21:16.437120453 (540h52m16.308203488s) [noise=true , address=145.79.227.15:4873 , piece_id=GNIWYTP5XP27MVY4FSIYJJVFYUJDW35SHHDPAHFYGUIUGIDJA6XQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437180547 (540h52m16.308263582s) [noise=true , address=77.90.31.11:4554 , piece_id=3JONTVGOFV7RTS7KLX5X6MERPIZYZBU3XTDSPXMPIPGLPMJCFFBQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437637788 (540h52m16.308720833s) [noise=true , address=145.79.227.18:4006 , piece_id=4SS2L6357KRRGBPJ76DHZSG3ZRZ7Y3HJMGXIZVZX2T6GQIZCBCRA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437649800 (540h52m16.308732845s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438090199 (540h52m16.309173234s) [noise=true , address=145.79.227.41:4086 , piece_id=C7LIFGRNHRDDZ367ZSLU2S2CXPU7CZDZSXSOLJHCU3THHEU72G4Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438136036 (540h52m16.309219071s) [noise=true , address=145.79.227.50:4374 , piece_id=DDW6NYPTSMB6BDDSE4CEZGJ2MTUG5KD56UC6YOBRKUMNN7SUFF2A, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438465745 (540h52m16.30954879s) [noise=true , address=145.79.227.19:4078 , piece_id=3DCEAV37L4TG6F3QJCIOKBUXLB2FINJ5KO5NSHZDLBTL4XC4WVNQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438516972 (540h52m16.309600017s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438938584 (540h52m16.310021619s) [noise=true , address=145.79.227.14:4840 , piece_id=I6KJU4LKWKSYXJH6AY46JTYLJKW5ERJZLQLM4PXOQSQBJAHOG6UA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438958372 (540h52m16.310041408s) [noise=true , address=45.91.116.5:4298 , piece_id=FBYW3JRQ4SWIK3O4BDA2WNAEEMIIAOVFVATRRHOQD3VBXZWUSKKA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439266469 (540h52m16.310349504s) [noise=true , address=45.91.116.8:4046 , piece_id=43MJGALQ4FPPRGVDHB5GOO55ND6W767IHEU7OJELLD55UH27JPJA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439308659 (540h52m16.310391704s) [noise=true , address=145.79.227.47:4266 , piece_id=LEIDMPE2ZUBBA7WNWNNY6KY5K27KUPBSLIS7WZNTWCBFMLXBC75Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439682792 (540h52m16.310765837s) [noise=true , address=45.91.116.10:4084 , piece_id=LLBSQAVLQKFRJUCMHSIFYBHRFDQCCYYBFNONRCOIMESH2TUAMCYQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439688543 (540h52m16.310771588s) [noise=true , address=145.79.227.46:4603 , piece_id=H3UXFPSTI4S4L46WJZOYCD6VBS6Z5B7P3Y6YZ3FVOASCLPTG3NCQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.440017369 (540h52m16.311100404s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.441260576 (540h52m16.312343621s) [noise=true , address=145.79.227.36:4517 , piece_id=E7OLX2EQ7QPJ7EKK3Z7JOLBZXHYZCUDN3IGAUAARAT3P6PGQQKJA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:18.054712406 (540h52m17.925795451s) [upload=474335]: Next returned nil segment Dec 28 04:21:18.054728186 (540h52m17.925811231s) [upload=474335]: Waiting for error group managing segments... Dec 28 04:21:18.441530576 (540h52m18.312613611s) [noise=true , address=145.79.227.46:4603 , piece_id=H3UXFPSTI4S4L46WJZOYCD6VBS6Z5B7P3Y6YZ3FVOASCLPTG3NCQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.448533951 (540h52m18.319616986s) [noise=true , address=145.79.227.41:4086 , piece_id=C7LIFGRNHRDDZ367ZSLU2S2CXPU7CZDZSXSOLJHCU3THHEU72G4Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.462572321 (540h52m18.333655366s) [noise=true , address=145.79.227.18:4006 , piece_id=4SS2L6357KRRGBPJ76DHZSG3ZRZ7Y3HJMGXIZVZX2T6GQIZCBCRA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.464562901 (540h52m18.335645947s) [noise=true , address=145.79.227.47:4266 , piece_id=LEIDMPE2ZUBBA7WNWNNY6KY5K27KUPBSLIS7WZNTWCBFMLXBC75Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.467915185 (540h52m18.33899823s) [noise=true , address=145.79.227.14:4840 , piece_id=I6KJU4LKWKSYXJH6AY46JTYLJKW5ERJZLQLM4PXOQSQBJAHOG6UA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.472609683 (540h52m18.343692718s) [noise=true , address=145.79.227.50:4374 , piece_id=DDW6NYPTSMB6BDDSE4CEZGJ2MTUG5KD56UC6YOBRKUMNN7SUFF2A, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.480745123 (540h52m18.351828168s) [noise=true , address=145.79.227.36:4517 , piece_id=E7OLX2EQ7QPJ7EKK3Z7JOLBZXHYZCUDN3IGAUAARAT3P6PGQQKJA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.488197813 (540h52m18.359280849s) [noise=true , address=145.79.227.15:4873 , piece_id=GNIWYTP5XP27MVY4FSIYJJVFYUJDW35SHHDPAHFYGUIUGIDJA6XQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.545421992 (540h52m18.416505037s) [noise=true , address=145.79.227.19:4078 , piece_id=3DCEAV37L4TG6F3QJCIOKBUXLB2FINJ5KO5NSHZDLBTL4XC4WVNQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.645863771 (540h52m18.516946817s) [noise=true , address=45.91.116.8:4046 , piece_id=43MJGALQ4FPPRGVDHB5GOO55ND6W767IHEU7OJELLD55UH27JPJA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.704987216 (540h52m18.576070251s) [noise=true , address=45.91.116.10:4084 , piece_id=LLBSQAVLQKFRJUCMHSIFYBHRFDQCCYYBFNONRCOIMESH2TUAMCYQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.717995325 (540h52m18.58907837s) [noise=true , address=77.90.31.11:4554 , piece_id=3JONTVGOFV7RTS7KLX5X6MERPIZYZBU3XTDSPXMPIPGLPMJCFFBQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.814309705 (540h52m18.68539275s) [noise=true , address=45.91.116.5:4298 , piece_id=FBYW3JRQ4SWIK3O4BDA2WNAEEMIIAOVFVATRRHOQD3VBXZWUSKKA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.814328781 (540h52m18.685411816s) [seg_pos={0 0}, upload=474335]: Segment reached optimal threshold of 13 pieces. Dec 28 04:21:18.814579217 (540h52m18.685662252s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:52950->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:21:18.814585580 (540h52m18.685668615s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.814837089 (540h52m18.685920144s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 04:21:18.814858549 (540h52m18.685941594s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.815007834 (540h52m18.686090879s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:51332->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:21:18.815017242 (540h52m18.686100287s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.815082696 (540h52m18.686165732s) [seg_pos={0 0}, upload=474335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:21:18.815164993 (540h52m18.686248028s) [upload=474335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:21:18.815169401 (540h52m18.686252436s) [upload=474335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:21:19.112094479 (540h52m18.983177524s) [upload=474335]: Done uploading segments... Dec 28 04:21:19.112116741 (540h52m18.983199776s) [upload=474335]: Upload finished. err: <nil> Duration: 6.253948365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X....] duration: 7.227543163s trace: https://jaeger.dev.storj.io/trace/52163586ab14fc00 FAIL (2025-12-28 04:24:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:21:16.187852237 (540h52m16.058935273s) [upload=474335]: Starting upload Dec 28 04:21:16.187919866 (540h52m16.059002901s) [upload=474335]: Done starting upload Dec 28 04:21:16.188001722 (540h52m16.059084767s) [upload=474335]: Uploading segments... Dec 28 04:21:16.188077466 (540h52m16.059160501s) [upload=474335]: Got next segment. Inline: false Dec 28 04:21:16.188113464 (540h52m16.05919658s) [upload=474335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:21:16.188121280 (540h52m16.059204325s) [upload=474335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:21:16.437024901 (540h52m16.308107956s) [seg_pos={0 0}, upload=474335]: Begin upload segment... Dec 28 04:21:16.437087139 (540h52m16.308170184s) [seg_pos={0 0}, upload=474335]: Done begin upload segment. Dec 28 04:21:16.437120453 (540h52m16.308203488s) [noise=true , address=145.79.227.15:4873 , piece_id=GNIWYTP5XP27MVY4FSIYJJVFYUJDW35SHHDPAHFYGUIUGIDJA6XQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437180547 (540h52m16.308263582s) [noise=true , address=77.90.31.11:4554 , piece_id=3JONTVGOFV7RTS7KLX5X6MERPIZYZBU3XTDSPXMPIPGLPMJCFFBQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437637788 (540h52m16.308720833s) [noise=true , address=145.79.227.18:4006 , piece_id=4SS2L6357KRRGBPJ76DHZSG3ZRZ7Y3HJMGXIZVZX2T6GQIZCBCRA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437649800 (540h52m16.308732845s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438090199 (540h52m16.309173234s) [noise=true , address=145.79.227.41:4086 , piece_id=C7LIFGRNHRDDZ367ZSLU2S2CXPU7CZDZSXSOLJHCU3THHEU72G4Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438136036 (540h52m16.309219071s) [noise=true , address=145.79.227.50:4374 , piece_id=DDW6NYPTSMB6BDDSE4CEZGJ2MTUG5KD56UC6YOBRKUMNN7SUFF2A, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438465745 (540h52m16.30954879s) [noise=true , address=145.79.227.19:4078 , piece_id=3DCEAV37L4TG6F3QJCIOKBUXLB2FINJ5KO5NSHZDLBTL4XC4WVNQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438516972 (540h52m16.309600017s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438938584 (540h52m16.310021619s) [noise=true , address=145.79.227.14:4840 , piece_id=I6KJU4LKWKSYXJH6AY46JTYLJKW5ERJZLQLM4PXOQSQBJAHOG6UA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438958372 (540h52m16.310041408s) [noise=true , address=45.91.116.5:4298 , piece_id=FBYW3JRQ4SWIK3O4BDA2WNAEEMIIAOVFVATRRHOQD3VBXZWUSKKA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439266469 (540h52m16.310349504s) [noise=true , address=45.91.116.8:4046 , piece_id=43MJGALQ4FPPRGVDHB5GOO55ND6W767IHEU7OJELLD55UH27JPJA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439308659 (540h52m16.310391704s) [noise=true , address=145.79.227.47:4266 , piece_id=LEIDMPE2ZUBBA7WNWNNY6KY5K27KUPBSLIS7WZNTWCBFMLXBC75Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439682792 (540h52m16.310765837s) [noise=true , address=45.91.116.10:4084 , piece_id=LLBSQAVLQKFRJUCMHSIFYBHRFDQCCYYBFNONRCOIMESH2TUAMCYQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439688543 (540h52m16.310771588s) [noise=true , address=145.79.227.46:4603 , piece_id=H3UXFPSTI4S4L46WJZOYCD6VBS6Z5B7P3Y6YZ3FVOASCLPTG3NCQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.440017369 (540h52m16.311100404s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.441260576 (540h52m16.312343621s) [noise=true , address=145.79.227.36:4517 , piece_id=E7OLX2EQ7QPJ7EKK3Z7JOLBZXHYZCUDN3IGAUAARAT3P6PGQQKJA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:18.054712406 (540h52m17.925795451s) [upload=474335]: Next returned nil segment Dec 28 04:21:18.054728186 (540h52m17.925811231s) [upload=474335]: Waiting for error group managing segments... Dec 28 04:21:18.441530576 (540h52m18.312613611s) [noise=true , address=145.79.227.46:4603 , piece_id=H3UXFPSTI4S4L46WJZOYCD6VBS6Z5B7P3Y6YZ3FVOASCLPTG3NCQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.448533951 (540h52m18.319616986s) [noise=true , address=145.79.227.41:4086 , piece_id=C7LIFGRNHRDDZ367ZSLU2S2CXPU7CZDZSXSOLJHCU3THHEU72G4Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.462572321 (540h52m18.333655366s) [noise=true , address=145.79.227.18:4006 , piece_id=4SS2L6357KRRGBPJ76DHZSG3ZRZ7Y3HJMGXIZVZX2T6GQIZCBCRA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.464562901 (540h52m18.335645947s) [noise=true , address=145.79.227.47:4266 , piece_id=LEIDMPE2ZUBBA7WNWNNY6KY5K27KUPBSLIS7WZNTWCBFMLXBC75Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.467915185 (540h52m18.33899823s) [noise=true , address=145.79.227.14:4840 , piece_id=I6KJU4LKWKSYXJH6AY46JTYLJKW5ERJZLQLM4PXOQSQBJAHOG6UA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.472609683 (540h52m18.343692718s) [noise=true , address=145.79.227.50:4374 , piece_id=DDW6NYPTSMB6BDDSE4CEZGJ2MTUG5KD56UC6YOBRKUMNN7SUFF2A, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.480745123 (540h52m18.351828168s) [noise=true , address=145.79.227.36:4517 , piece_id=E7OLX2EQ7QPJ7EKK3Z7JOLBZXHYZCUDN3IGAUAARAT3P6PGQQKJA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.488197813 (540h52m18.359280849s) [noise=true , address=145.79.227.15:4873 , piece_id=GNIWYTP5XP27MVY4FSIYJJVFYUJDW35SHHDPAHFYGUIUGIDJA6XQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.545421992 (540h52m18.416505037s) [noise=true , address=145.79.227.19:4078 , piece_id=3DCEAV37L4TG6F3QJCIOKBUXLB2FINJ5KO5NSHZDLBTL4XC4WVNQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.645863771 (540h52m18.516946817s) [noise=true , address=45.91.116.8:4046 , piece_id=43MJGALQ4FPPRGVDHB5GOO55ND6W767IHEU7OJELLD55UH27JPJA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.704987216 (540h52m18.576070251s) [noise=true , address=45.91.116.10:4084 , piece_id=LLBSQAVLQKFRJUCMHSIFYBHRFDQCCYYBFNONRCOIMESH2TUAMCYQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.717995325 (540h52m18.58907837s) [noise=true , address=77.90.31.11:4554 , piece_id=3JONTVGOFV7RTS7KLX5X6MERPIZYZBU3XTDSPXMPIPGLPMJCFFBQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.814309705 (540h52m18.68539275s) [noise=true , address=45.91.116.5:4298 , piece_id=FBYW3JRQ4SWIK3O4BDA2WNAEEMIIAOVFVATRRHOQD3VBXZWUSKKA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.814328781 (540h52m18.685411816s) [seg_pos={0 0}, upload=474335]: Segment reached optimal threshold of 13 pieces. Dec 28 04:21:18.814579217 (540h52m18.685662252s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:52950->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:21:18.814585580 (540h52m18.685668615s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.814837089 (540h52m18.685920144s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 04:21:18.814858549 (540h52m18.685941594s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.815007834 (540h52m18.686090879s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:51332->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:21:18.815017242 (540h52m18.686100287s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.815082696 (540h52m18.686165732s) [seg_pos={0 0}, upload=474335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:21:18.815164993 (540h52m18.686248028s) [upload=474335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:21:18.815169401 (540h52m18.686252436s) [upload=474335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:21:19.112094479 (540h52m18.983177524s) [upload=474335]: Done uploading segments... Dec 28 04:21:19.112116741 (540h52m18.983199776s) [upload=474335]: Upload finished. err: <nil> Duration: 7.227568622s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X......X................] duration: 24.761385867s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:24:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 24.761400385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [...X..X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX....XX] duration: 5.000898359s trace: https://jaeger.dev.storj.io/trace/2a1a75884003581b FAIL (2025-12-28 04:24:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0009202s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 44 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X...] duration: 6.132119735s trace: https://jaeger.dev.storj.io/trace/52163586ab14fc00 FAIL (2025-12-28 04:24:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:21:16.187852237 (540h52m16.058935273s) [upload=474335]: Starting upload Dec 28 04:21:16.187919866 (540h52m16.059002901s) [upload=474335]: Done starting upload Dec 28 04:21:16.188001722 (540h52m16.059084767s) [upload=474335]: Uploading segments... Dec 28 04:21:16.188077466 (540h52m16.059160501s) [upload=474335]: Got next segment. Inline: false Dec 28 04:21:16.188113464 (540h52m16.05919658s) [upload=474335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:21:16.188121280 (540h52m16.059204325s) [upload=474335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:21:16.437024901 (540h52m16.308107956s) [seg_pos={0 0}, upload=474335]: Begin upload segment... Dec 28 04:21:16.437087139 (540h52m16.308170184s) [seg_pos={0 0}, upload=474335]: Done begin upload segment. Dec 28 04:21:16.437120453 (540h52m16.308203488s) [noise=true , address=145.79.227.15:4873 , piece_id=GNIWYTP5XP27MVY4FSIYJJVFYUJDW35SHHDPAHFYGUIUGIDJA6XQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437180547 (540h52m16.308263582s) [noise=true , address=77.90.31.11:4554 , piece_id=3JONTVGOFV7RTS7KLX5X6MERPIZYZBU3XTDSPXMPIPGLPMJCFFBQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437637788 (540h52m16.308720833s) [noise=true , address=145.79.227.18:4006 , piece_id=4SS2L6357KRRGBPJ76DHZSG3ZRZ7Y3HJMGXIZVZX2T6GQIZCBCRA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437649800 (540h52m16.308732845s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438090199 (540h52m16.309173234s) [noise=true , address=145.79.227.41:4086 , piece_id=C7LIFGRNHRDDZ367ZSLU2S2CXPU7CZDZSXSOLJHCU3THHEU72G4Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438136036 (540h52m16.309219071s) [noise=true , address=145.79.227.50:4374 , piece_id=DDW6NYPTSMB6BDDSE4CEZGJ2MTUG5KD56UC6YOBRKUMNN7SUFF2A, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438465745 (540h52m16.30954879s) [noise=true , address=145.79.227.19:4078 , piece_id=3DCEAV37L4TG6F3QJCIOKBUXLB2FINJ5KO5NSHZDLBTL4XC4WVNQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438516972 (540h52m16.309600017s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438938584 (540h52m16.310021619s) [noise=true , address=145.79.227.14:4840 , piece_id=I6KJU4LKWKSYXJH6AY46JTYLJKW5ERJZLQLM4PXOQSQBJAHOG6UA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438958372 (540h52m16.310041408s) [noise=true , address=45.91.116.5:4298 , piece_id=FBYW3JRQ4SWIK3O4BDA2WNAEEMIIAOVFVATRRHOQD3VBXZWUSKKA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439266469 (540h52m16.310349504s) [noise=true , address=45.91.116.8:4046 , piece_id=43MJGALQ4FPPRGVDHB5GOO55ND6W767IHEU7OJELLD55UH27JPJA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439308659 (540h52m16.310391704s) [noise=true , address=145.79.227.47:4266 , piece_id=LEIDMPE2ZUBBA7WNWNNY6KY5K27KUPBSLIS7WZNTWCBFMLXBC75Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439682792 (540h52m16.310765837s) [noise=true , address=45.91.116.10:4084 , piece_id=LLBSQAVLQKFRJUCMHSIFYBHRFDQCCYYBFNONRCOIMESH2TUAMCYQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439688543 (540h52m16.310771588s) [noise=true , address=145.79.227.46:4603 , piece_id=H3UXFPSTI4S4L46WJZOYCD6VBS6Z5B7P3Y6YZ3FVOASCLPTG3NCQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.440017369 (540h52m16.311100404s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.441260576 (540h52m16.312343621s) [noise=true , address=145.79.227.36:4517 , piece_id=E7OLX2EQ7QPJ7EKK3Z7JOLBZXHYZCUDN3IGAUAARAT3P6PGQQKJA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:18.054712406 (540h52m17.925795451s) [upload=474335]: Next returned nil segment Dec 28 04:21:18.054728186 (540h52m17.925811231s) [upload=474335]: Waiting for error group managing segments... Dec 28 04:21:18.441530576 (540h52m18.312613611s) [noise=true , address=145.79.227.46:4603 , piece_id=H3UXFPSTI4S4L46WJZOYCD6VBS6Z5B7P3Y6YZ3FVOASCLPTG3NCQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.448533951 (540h52m18.319616986s) [noise=true , address=145.79.227.41:4086 , piece_id=C7LIFGRNHRDDZ367ZSLU2S2CXPU7CZDZSXSOLJHCU3THHEU72G4Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.462572321 (540h52m18.333655366s) [noise=true , address=145.79.227.18:4006 , piece_id=4SS2L6357KRRGBPJ76DHZSG3ZRZ7Y3HJMGXIZVZX2T6GQIZCBCRA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.464562901 (540h52m18.335645947s) [noise=true , address=145.79.227.47:4266 , piece_id=LEIDMPE2ZUBBA7WNWNNY6KY5K27KUPBSLIS7WZNTWCBFMLXBC75Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.467915185 (540h52m18.33899823s) [noise=true , address=145.79.227.14:4840 , piece_id=I6KJU4LKWKSYXJH6AY46JTYLJKW5ERJZLQLM4PXOQSQBJAHOG6UA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.472609683 (540h52m18.343692718s) [noise=true , address=145.79.227.50:4374 , piece_id=DDW6NYPTSMB6BDDSE4CEZGJ2MTUG5KD56UC6YOBRKUMNN7SUFF2A, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.480745123 (540h52m18.351828168s) [noise=true , address=145.79.227.36:4517 , piece_id=E7OLX2EQ7QPJ7EKK3Z7JOLBZXHYZCUDN3IGAUAARAT3P6PGQQKJA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.488197813 (540h52m18.359280849s) [noise=true , address=145.79.227.15:4873 , piece_id=GNIWYTP5XP27MVY4FSIYJJVFYUJDW35SHHDPAHFYGUIUGIDJA6XQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.545421992 (540h52m18.416505037s) [noise=true , address=145.79.227.19:4078 , piece_id=3DCEAV37L4TG6F3QJCIOKBUXLB2FINJ5KO5NSHZDLBTL4XC4WVNQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.645863771 (540h52m18.516946817s) [noise=true , address=45.91.116.8:4046 , piece_id=43MJGALQ4FPPRGVDHB5GOO55ND6W767IHEU7OJELLD55UH27JPJA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.704987216 (540h52m18.576070251s) [noise=true , address=45.91.116.10:4084 , piece_id=LLBSQAVLQKFRJUCMHSIFYBHRFDQCCYYBFNONRCOIMESH2TUAMCYQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.717995325 (540h52m18.58907837s) [noise=true , address=77.90.31.11:4554 , piece_id=3JONTVGOFV7RTS7KLX5X6MERPIZYZBU3XTDSPXMPIPGLPMJCFFBQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.814309705 (540h52m18.68539275s) [noise=true , address=45.91.116.5:4298 , piece_id=FBYW3JRQ4SWIK3O4BDA2WNAEEMIIAOVFVATRRHOQD3VBXZWUSKKA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.814328781 (540h52m18.685411816s) [seg_pos={0 0}, upload=474335]: Segment reached optimal threshold of 13 pieces. Dec 28 04:21:18.814579217 (540h52m18.685662252s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:52950->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:21:18.814585580 (540h52m18.685668615s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.814837089 (540h52m18.685920144s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 04:21:18.814858549 (540h52m18.685941594s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.815007834 (540h52m18.686090879s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:51332->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:21:18.815017242 (540h52m18.686100287s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.815082696 (540h52m18.686165732s) [seg_pos={0 0}, upload=474335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:21:18.815164993 (540h52m18.686248028s) [upload=474335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:21:18.815169401 (540h52m18.686252436s) [upload=474335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:21:19.112094479 (540h52m18.983177524s) [upload=474335]: Done uploading segments... Dec 28 04:21:19.112116741 (540h52m18.983199776s) [upload=474335]: Upload finished. err: <nil> Duration: 6.132149411s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X......X...............] duration: 28.437405968s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:23:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 28.4374332s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X...X..X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX....X] duration: 5.000913686s trace: https://jaeger.dev.storj.io/trace/509419a9cfb8b0de FAIL (2025-12-28 04:23:47 UTC) orbiter warning: context deadline exceeded Duration: 5.000938053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X..] duration: 19.123454209s trace: https://jaeger.dev.storj.io/trace/52163586ab14fc00 FAIL (2025-12-28 04:23:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:21:16.187852237 (540h52m16.058935273s) [upload=474335]: Starting upload Dec 28 04:21:16.187919866 (540h52m16.059002901s) [upload=474335]: Done starting upload Dec 28 04:21:16.188001722 (540h52m16.059084767s) [upload=474335]: Uploading segments... Dec 28 04:21:16.188077466 (540h52m16.059160501s) [upload=474335]: Got next segment. Inline: false Dec 28 04:21:16.188113464 (540h52m16.05919658s) [upload=474335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:21:16.188121280 (540h52m16.059204325s) [upload=474335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:21:16.437024901 (540h52m16.308107956s) [seg_pos={0 0}, upload=474335]: Begin upload segment... Dec 28 04:21:16.437087139 (540h52m16.308170184s) [seg_pos={0 0}, upload=474335]: Done begin upload segment. Dec 28 04:21:16.437120453 (540h52m16.308203488s) [noise=true , address=145.79.227.15:4873 , piece_id=GNIWYTP5XP27MVY4FSIYJJVFYUJDW35SHHDPAHFYGUIUGIDJA6XQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437180547 (540h52m16.308263582s) [noise=true , address=77.90.31.11:4554 , piece_id=3JONTVGOFV7RTS7KLX5X6MERPIZYZBU3XTDSPXMPIPGLPMJCFFBQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437637788 (540h52m16.308720833s) [noise=true , address=145.79.227.18:4006 , piece_id=4SS2L6357KRRGBPJ76DHZSG3ZRZ7Y3HJMGXIZVZX2T6GQIZCBCRA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437649800 (540h52m16.308732845s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438090199 (540h52m16.309173234s) [noise=true , address=145.79.227.41:4086 , piece_id=C7LIFGRNHRDDZ367ZSLU2S2CXPU7CZDZSXSOLJHCU3THHEU72G4Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438136036 (540h52m16.309219071s) [noise=true , address=145.79.227.50:4374 , piece_id=DDW6NYPTSMB6BDDSE4CEZGJ2MTUG5KD56UC6YOBRKUMNN7SUFF2A, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438465745 (540h52m16.30954879s) [noise=true , address=145.79.227.19:4078 , piece_id=3DCEAV37L4TG6F3QJCIOKBUXLB2FINJ5KO5NSHZDLBTL4XC4WVNQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438516972 (540h52m16.309600017s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438938584 (540h52m16.310021619s) [noise=true , address=145.79.227.14:4840 , piece_id=I6KJU4LKWKSYXJH6AY46JTYLJKW5ERJZLQLM4PXOQSQBJAHOG6UA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438958372 (540h52m16.310041408s) [noise=true , address=45.91.116.5:4298 , piece_id=FBYW3JRQ4SWIK3O4BDA2WNAEEMIIAOVFVATRRHOQD3VBXZWUSKKA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439266469 (540h52m16.310349504s) [noise=true , address=45.91.116.8:4046 , piece_id=43MJGALQ4FPPRGVDHB5GOO55ND6W767IHEU7OJELLD55UH27JPJA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439308659 (540h52m16.310391704s) [noise=true , address=145.79.227.47:4266 , piece_id=LEIDMPE2ZUBBA7WNWNNY6KY5K27KUPBSLIS7WZNTWCBFMLXBC75Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439682792 (540h52m16.310765837s) [noise=true , address=45.91.116.10:4084 , piece_id=LLBSQAVLQKFRJUCMHSIFYBHRFDQCCYYBFNONRCOIMESH2TUAMCYQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439688543 (540h52m16.310771588s) [noise=true , address=145.79.227.46:4603 , piece_id=H3UXFPSTI4S4L46WJZOYCD6VBS6Z5B7P3Y6YZ3FVOASCLPTG3NCQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.440017369 (540h52m16.311100404s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.441260576 (540h52m16.312343621s) [noise=true , address=145.79.227.36:4517 , piece_id=E7OLX2EQ7QPJ7EKK3Z7JOLBZXHYZCUDN3IGAUAARAT3P6PGQQKJA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:18.054712406 (540h52m17.925795451s) [upload=474335]: Next returned nil segment Dec 28 04:21:18.054728186 (540h52m17.925811231s) [upload=474335]: Waiting for error group managing segments... Dec 28 04:21:18.441530576 (540h52m18.312613611s) [noise=true , address=145.79.227.46:4603 , piece_id=H3UXFPSTI4S4L46WJZOYCD6VBS6Z5B7P3Y6YZ3FVOASCLPTG3NCQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.448533951 (540h52m18.319616986s) [noise=true , address=145.79.227.41:4086 , piece_id=C7LIFGRNHRDDZ367ZSLU2S2CXPU7CZDZSXSOLJHCU3THHEU72G4Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.462572321 (540h52m18.333655366s) [noise=true , address=145.79.227.18:4006 , piece_id=4SS2L6357KRRGBPJ76DHZSG3ZRZ7Y3HJMGXIZVZX2T6GQIZCBCRA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.464562901 (540h52m18.335645947s) [noise=true , address=145.79.227.47:4266 , piece_id=LEIDMPE2ZUBBA7WNWNNY6KY5K27KUPBSLIS7WZNTWCBFMLXBC75Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.467915185 (540h52m18.33899823s) [noise=true , address=145.79.227.14:4840 , piece_id=I6KJU4LKWKSYXJH6AY46JTYLJKW5ERJZLQLM4PXOQSQBJAHOG6UA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.472609683 (540h52m18.343692718s) [noise=true , address=145.79.227.50:4374 , piece_id=DDW6NYPTSMB6BDDSE4CEZGJ2MTUG5KD56UC6YOBRKUMNN7SUFF2A, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.480745123 (540h52m18.351828168s) [noise=true , address=145.79.227.36:4517 , piece_id=E7OLX2EQ7QPJ7EKK3Z7JOLBZXHYZCUDN3IGAUAARAT3P6PGQQKJA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.488197813 (540h52m18.359280849s) [noise=true , address=145.79.227.15:4873 , piece_id=GNIWYTP5XP27MVY4FSIYJJVFYUJDW35SHHDPAHFYGUIUGIDJA6XQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.545421992 (540h52m18.416505037s) [noise=true , address=145.79.227.19:4078 , piece_id=3DCEAV37L4TG6F3QJCIOKBUXLB2FINJ5KO5NSHZDLBTL4XC4WVNQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.645863771 (540h52m18.516946817s) [noise=true , address=45.91.116.8:4046 , piece_id=43MJGALQ4FPPRGVDHB5GOO55ND6W767IHEU7OJELLD55UH27JPJA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.704987216 (540h52m18.576070251s) [noise=true , address=45.91.116.10:4084 , piece_id=LLBSQAVLQKFRJUCMHSIFYBHRFDQCCYYBFNONRCOIMESH2TUAMCYQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.717995325 (540h52m18.58907837s) [noise=true , address=77.90.31.11:4554 , piece_id=3JONTVGOFV7RTS7KLX5X6MERPIZYZBU3XTDSPXMPIPGLPMJCFFBQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.814309705 (540h52m18.68539275s) [noise=true , address=45.91.116.5:4298 , piece_id=FBYW3JRQ4SWIK3O4BDA2WNAEEMIIAOVFVATRRHOQD3VBXZWUSKKA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.814328781 (540h52m18.685411816s) [seg_pos={0 0}, upload=474335]: Segment reached optimal threshold of 13 pieces. Dec 28 04:21:18.814579217 (540h52m18.685662252s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:52950->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:21:18.814585580 (540h52m18.685668615s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.814837089 (540h52m18.685920144s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 04:21:18.814858549 (540h52m18.685941594s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.815007834 (540h52m18.686090879s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:51332->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:21:18.815017242 (540h52m18.686100287s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.815082696 (540h52m18.686165732s) [seg_pos={0 0}, upload=474335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:21:18.815164993 (540h52m18.686248028s) [upload=474335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:21:18.815169401 (540h52m18.686252436s) [upload=474335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:21:19.112094479 (540h52m18.983177524s) [upload=474335]: Done uploading segments... Dec 28 04:21:19.112116741 (540h52m18.983199776s) [upload=474335]: Upload finished. err: <nil> Duration: 19.123482904s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 7.193059845s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:23:20 UTC) orbiter warning: context deadline exceeded Duration: 7.193090854s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X......X..............] duration: 33.390283261s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:23:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 33.390308217s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X...X..X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX....] duration: 3.484163283s trace: https://jaeger.dev.storj.io/trace/36c67681f69a29b0 FAIL (2025-12-28 04:22:59 UTC) orbiter warning: context deadline exceeded Duration: 3.484184572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X.] duration: 8.82159175s trace: https://jaeger.dev.storj.io/trace/52163586ab14fc00 FAIL (2025-12-28 04:22:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:21:16.187852237 (540h52m16.058935273s) [upload=474335]: Starting upload Dec 28 04:21:16.187919866 (540h52m16.059002901s) [upload=474335]: Done starting upload Dec 28 04:21:16.188001722 (540h52m16.059084767s) [upload=474335]: Uploading segments... Dec 28 04:21:16.188077466 (540h52m16.059160501s) [upload=474335]: Got next segment. Inline: false Dec 28 04:21:16.188113464 (540h52m16.05919658s) [upload=474335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:21:16.188121280 (540h52m16.059204325s) [upload=474335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:21:16.437024901 (540h52m16.308107956s) [seg_pos={0 0}, upload=474335]: Begin upload segment... Dec 28 04:21:16.437087139 (540h52m16.308170184s) [seg_pos={0 0}, upload=474335]: Done begin upload segment. Dec 28 04:21:16.437120453 (540h52m16.308203488s) [noise=true , address=145.79.227.15:4873 , piece_id=GNIWYTP5XP27MVY4FSIYJJVFYUJDW35SHHDPAHFYGUIUGIDJA6XQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437180547 (540h52m16.308263582s) [noise=true , address=77.90.31.11:4554 , piece_id=3JONTVGOFV7RTS7KLX5X6MERPIZYZBU3XTDSPXMPIPGLPMJCFFBQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437637788 (540h52m16.308720833s) [noise=true , address=145.79.227.18:4006 , piece_id=4SS2L6357KRRGBPJ76DHZSG3ZRZ7Y3HJMGXIZVZX2T6GQIZCBCRA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437649800 (540h52m16.308732845s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438090199 (540h52m16.309173234s) [noise=true , address=145.79.227.41:4086 , piece_id=C7LIFGRNHRDDZ367ZSLU2S2CXPU7CZDZSXSOLJHCU3THHEU72G4Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438136036 (540h52m16.309219071s) [noise=true , address=145.79.227.50:4374 , piece_id=DDW6NYPTSMB6BDDSE4CEZGJ2MTUG5KD56UC6YOBRKUMNN7SUFF2A, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438465745 (540h52m16.30954879s) [noise=true , address=145.79.227.19:4078 , piece_id=3DCEAV37L4TG6F3QJCIOKBUXLB2FINJ5KO5NSHZDLBTL4XC4WVNQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438516972 (540h52m16.309600017s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438938584 (540h52m16.310021619s) [noise=true , address=145.79.227.14:4840 , piece_id=I6KJU4LKWKSYXJH6AY46JTYLJKW5ERJZLQLM4PXOQSQBJAHOG6UA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438958372 (540h52m16.310041408s) [noise=true , address=45.91.116.5:4298 , piece_id=FBYW3JRQ4SWIK3O4BDA2WNAEEMIIAOVFVATRRHOQD3VBXZWUSKKA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439266469 (540h52m16.310349504s) [noise=true , address=45.91.116.8:4046 , piece_id=43MJGALQ4FPPRGVDHB5GOO55ND6W767IHEU7OJELLD55UH27JPJA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439308659 (540h52m16.310391704s) [noise=true , address=145.79.227.47:4266 , piece_id=LEIDMPE2ZUBBA7WNWNNY6KY5K27KUPBSLIS7WZNTWCBFMLXBC75Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439682792 (540h52m16.310765837s) [noise=true , address=45.91.116.10:4084 , piece_id=LLBSQAVLQKFRJUCMHSIFYBHRFDQCCYYBFNONRCOIMESH2TUAMCYQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439688543 (540h52m16.310771588s) [noise=true , address=145.79.227.46:4603 , piece_id=H3UXFPSTI4S4L46WJZOYCD6VBS6Z5B7P3Y6YZ3FVOASCLPTG3NCQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.440017369 (540h52m16.311100404s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.441260576 (540h52m16.312343621s) [noise=true , address=145.79.227.36:4517 , piece_id=E7OLX2EQ7QPJ7EKK3Z7JOLBZXHYZCUDN3IGAUAARAT3P6PGQQKJA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:18.054712406 (540h52m17.925795451s) [upload=474335]: Next returned nil segment Dec 28 04:21:18.054728186 (540h52m17.925811231s) [upload=474335]: Waiting for error group managing segments... Dec 28 04:21:18.441530576 (540h52m18.312613611s) [noise=true , address=145.79.227.46:4603 , piece_id=H3UXFPSTI4S4L46WJZOYCD6VBS6Z5B7P3Y6YZ3FVOASCLPTG3NCQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.448533951 (540h52m18.319616986s) [noise=true , address=145.79.227.41:4086 , piece_id=C7LIFGRNHRDDZ367ZSLU2S2CXPU7CZDZSXSOLJHCU3THHEU72G4Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.462572321 (540h52m18.333655366s) [noise=true , address=145.79.227.18:4006 , piece_id=4SS2L6357KRRGBPJ76DHZSG3ZRZ7Y3HJMGXIZVZX2T6GQIZCBCRA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.464562901 (540h52m18.335645947s) [noise=true , address=145.79.227.47:4266 , piece_id=LEIDMPE2ZUBBA7WNWNNY6KY5K27KUPBSLIS7WZNTWCBFMLXBC75Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.467915185 (540h52m18.33899823s) [noise=true , address=145.79.227.14:4840 , piece_id=I6KJU4LKWKSYXJH6AY46JTYLJKW5ERJZLQLM4PXOQSQBJAHOG6UA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.472609683 (540h52m18.343692718s) [noise=true , address=145.79.227.50:4374 , piece_id=DDW6NYPTSMB6BDDSE4CEZGJ2MTUG5KD56UC6YOBRKUMNN7SUFF2A, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.480745123 (540h52m18.351828168s) [noise=true , address=145.79.227.36:4517 , piece_id=E7OLX2EQ7QPJ7EKK3Z7JOLBZXHYZCUDN3IGAUAARAT3P6PGQQKJA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.488197813 (540h52m18.359280849s) [noise=true , address=145.79.227.15:4873 , piece_id=GNIWYTP5XP27MVY4FSIYJJVFYUJDW35SHHDPAHFYGUIUGIDJA6XQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.545421992 (540h52m18.416505037s) [noise=true , address=145.79.227.19:4078 , piece_id=3DCEAV37L4TG6F3QJCIOKBUXLB2FINJ5KO5NSHZDLBTL4XC4WVNQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.645863771 (540h52m18.516946817s) [noise=true , address=45.91.116.8:4046 , piece_id=43MJGALQ4FPPRGVDHB5GOO55ND6W767IHEU7OJELLD55UH27JPJA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.704987216 (540h52m18.576070251s) [noise=true , address=45.91.116.10:4084 , piece_id=LLBSQAVLQKFRJUCMHSIFYBHRFDQCCYYBFNONRCOIMESH2TUAMCYQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.717995325 (540h52m18.58907837s) [noise=true , address=77.90.31.11:4554 , piece_id=3JONTVGOFV7RTS7KLX5X6MERPIZYZBU3XTDSPXMPIPGLPMJCFFBQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.814309705 (540h52m18.68539275s) [noise=true , address=45.91.116.5:4298 , piece_id=FBYW3JRQ4SWIK3O4BDA2WNAEEMIIAOVFVATRRHOQD3VBXZWUSKKA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.814328781 (540h52m18.685411816s) [seg_pos={0 0}, upload=474335]: Segment reached optimal threshold of 13 pieces. Dec 28 04:21:18.814579217 (540h52m18.685662252s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:52950->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:21:18.814585580 (540h52m18.685668615s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.814837089 (540h52m18.685920144s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 04:21:18.814858549 (540h52m18.685941594s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.815007834 (540h52m18.686090879s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:51332->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:21:18.815017242 (540h52m18.686100287s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.815082696 (540h52m18.686165732s) [seg_pos={0 0}, upload=474335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:21:18.815164993 (540h52m18.686248028s) [upload=474335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:21:18.815169401 (540h52m18.686252436s) [upload=474335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:21:19.112094479 (540h52m18.983177524s) [upload=474335]: Done uploading segments... Dec 28 04:21:19.112116741 (540h52m18.983199776s) [upload=474335]: Upload finished. err: <nil> Duration: 8.821616527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X......X.............] duration: 30.179432076s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:22:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 30.179450099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.178904652s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:22:17 UTC) orbiter warning: context deadline exceeded Duration: 6.178937803s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X......X............] duration: 19.862607059s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:21:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 19.862623931s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X.X...X..X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX...] duration: 3.623902275s trace: https://jaeger.dev.storj.io/trace/36c67681f69a29b0 FAIL (2025-12-28 04:21:51 UTC) orbiter warning: context deadline exceeded Duration: 3.623925769s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 8.681932096s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:21:46 UTC) orbiter warning: context deadline exceeded Duration: 8.681952064s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...X] duration: 30.004780172s trace: https://jaeger.dev.storj.io/trace/52163586ab14fc00 FAIL (2025-12-28 04:21:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:21:16.187852237 (540h52m16.058935273s) [upload=474335]: Starting upload Dec 28 04:21:16.187919866 (540h52m16.059002901s) [upload=474335]: Done starting upload Dec 28 04:21:16.188001722 (540h52m16.059084767s) [upload=474335]: Uploading segments... Dec 28 04:21:16.188077466 (540h52m16.059160501s) [upload=474335]: Got next segment. Inline: false Dec 28 04:21:16.188113464 (540h52m16.05919658s) [upload=474335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:21:16.188121280 (540h52m16.059204325s) [upload=474335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:21:16.437024901 (540h52m16.308107956s) [seg_pos={0 0}, upload=474335]: Begin upload segment... Dec 28 04:21:16.437087139 (540h52m16.308170184s) [seg_pos={0 0}, upload=474335]: Done begin upload segment. Dec 28 04:21:16.437120453 (540h52m16.308203488s) [noise=true , address=145.79.227.15:4873 , piece_id=GNIWYTP5XP27MVY4FSIYJJVFYUJDW35SHHDPAHFYGUIUGIDJA6XQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437180547 (540h52m16.308263582s) [noise=true , address=77.90.31.11:4554 , piece_id=3JONTVGOFV7RTS7KLX5X6MERPIZYZBU3XTDSPXMPIPGLPMJCFFBQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437637788 (540h52m16.308720833s) [noise=true , address=145.79.227.18:4006 , piece_id=4SS2L6357KRRGBPJ76DHZSG3ZRZ7Y3HJMGXIZVZX2T6GQIZCBCRA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.437649800 (540h52m16.308732845s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438090199 (540h52m16.309173234s) [noise=true , address=145.79.227.41:4086 , piece_id=C7LIFGRNHRDDZ367ZSLU2S2CXPU7CZDZSXSOLJHCU3THHEU72G4Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438136036 (540h52m16.309219071s) [noise=true , address=145.79.227.50:4374 , piece_id=DDW6NYPTSMB6BDDSE4CEZGJ2MTUG5KD56UC6YOBRKUMNN7SUFF2A, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438465745 (540h52m16.30954879s) [noise=true , address=145.79.227.19:4078 , piece_id=3DCEAV37L4TG6F3QJCIOKBUXLB2FINJ5KO5NSHZDLBTL4XC4WVNQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438516972 (540h52m16.309600017s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438938584 (540h52m16.310021619s) [noise=true , address=145.79.227.14:4840 , piece_id=I6KJU4LKWKSYXJH6AY46JTYLJKW5ERJZLQLM4PXOQSQBJAHOG6UA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.438958372 (540h52m16.310041408s) [noise=true , address=45.91.116.5:4298 , piece_id=FBYW3JRQ4SWIK3O4BDA2WNAEEMIIAOVFVATRRHOQD3VBXZWUSKKA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439266469 (540h52m16.310349504s) [noise=true , address=45.91.116.8:4046 , piece_id=43MJGALQ4FPPRGVDHB5GOO55ND6W767IHEU7OJELLD55UH27JPJA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439308659 (540h52m16.310391704s) [noise=true , address=145.79.227.47:4266 , piece_id=LEIDMPE2ZUBBA7WNWNNY6KY5K27KUPBSLIS7WZNTWCBFMLXBC75Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439682792 (540h52m16.310765837s) [noise=true , address=45.91.116.10:4084 , piece_id=LLBSQAVLQKFRJUCMHSIFYBHRFDQCCYYBFNONRCOIMESH2TUAMCYQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.439688543 (540h52m16.310771588s) [noise=true , address=145.79.227.46:4603 , piece_id=H3UXFPSTI4S4L46WJZOYCD6VBS6Z5B7P3Y6YZ3FVOASCLPTG3NCQ, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.440017369 (540h52m16.311100404s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:16.441260576 (540h52m16.312343621s) [noise=true , address=145.79.227.36:4517 , piece_id=E7OLX2EQ7QPJ7EKK3Z7JOLBZXHYZCUDN3IGAUAARAT3P6PGQQKJA, seg_pos={0 0}, upload=474335]: Uploading piece... Dec 28 04:21:18.054712406 (540h52m17.925795451s) [upload=474335]: Next returned nil segment Dec 28 04:21:18.054728186 (540h52m17.925811231s) [upload=474335]: Waiting for error group managing segments... Dec 28 04:21:18.441530576 (540h52m18.312613611s) [noise=true , address=145.79.227.46:4603 , piece_id=H3UXFPSTI4S4L46WJZOYCD6VBS6Z5B7P3Y6YZ3FVOASCLPTG3NCQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.448533951 (540h52m18.319616986s) [noise=true , address=145.79.227.41:4086 , piece_id=C7LIFGRNHRDDZ367ZSLU2S2CXPU7CZDZSXSOLJHCU3THHEU72G4Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.462572321 (540h52m18.333655366s) [noise=true , address=145.79.227.18:4006 , piece_id=4SS2L6357KRRGBPJ76DHZSG3ZRZ7Y3HJMGXIZVZX2T6GQIZCBCRA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.464562901 (540h52m18.335645947s) [noise=true , address=145.79.227.47:4266 , piece_id=LEIDMPE2ZUBBA7WNWNNY6KY5K27KUPBSLIS7WZNTWCBFMLXBC75Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.467915185 (540h52m18.33899823s) [noise=true , address=145.79.227.14:4840 , piece_id=I6KJU4LKWKSYXJH6AY46JTYLJKW5ERJZLQLM4PXOQSQBJAHOG6UA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.472609683 (540h52m18.343692718s) [noise=true , address=145.79.227.50:4374 , piece_id=DDW6NYPTSMB6BDDSE4CEZGJ2MTUG5KD56UC6YOBRKUMNN7SUFF2A, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.480745123 (540h52m18.351828168s) [noise=true , address=145.79.227.36:4517 , piece_id=E7OLX2EQ7QPJ7EKK3Z7JOLBZXHYZCUDN3IGAUAARAT3P6PGQQKJA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.488197813 (540h52m18.359280849s) [noise=true , address=145.79.227.15:4873 , piece_id=GNIWYTP5XP27MVY4FSIYJJVFYUJDW35SHHDPAHFYGUIUGIDJA6XQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.545421992 (540h52m18.416505037s) [noise=true , address=145.79.227.19:4078 , piece_id=3DCEAV37L4TG6F3QJCIOKBUXLB2FINJ5KO5NSHZDLBTL4XC4WVNQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.645863771 (540h52m18.516946817s) [noise=true , address=45.91.116.8:4046 , piece_id=43MJGALQ4FPPRGVDHB5GOO55ND6W767IHEU7OJELLD55UH27JPJA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.704987216 (540h52m18.576070251s) [noise=true , address=45.91.116.10:4084 , piece_id=LLBSQAVLQKFRJUCMHSIFYBHRFDQCCYYBFNONRCOIMESH2TUAMCYQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.717995325 (540h52m18.58907837s) [noise=true , address=77.90.31.11:4554 , piece_id=3JONTVGOFV7RTS7KLX5X6MERPIZYZBU3XTDSPXMPIPGLPMJCFFBQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.814309705 (540h52m18.68539275s) [noise=true , address=45.91.116.5:4298 , piece_id=FBYW3JRQ4SWIK3O4BDA2WNAEEMIIAOVFVATRRHOQD3VBXZWUSKKA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: <nil> Dec 28 04:21:18.814328781 (540h52m18.685411816s) [seg_pos={0 0}, upload=474335]: Segment reached optimal threshold of 13 pieces. Dec 28 04:21:18.814579217 (540h52m18.685662252s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:52950->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:21:18.814585580 (540h52m18.685668615s) [noise=true , address=45.91.116.13:4154 , piece_id=IIXR6BJTH2PWB6S7RKKSPXYV5DVTZRZ76DMS6RODOPUVGTN46V3Q, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.814837089 (540h52m18.685920144s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 04:21:18.814858549 (540h52m18.685941594s) [noise=true , address=77.90.31.12:4590 , piece_id=K3AMPCLQUSBCBWPNDRF4XWREUASERDBJSL6RTGPCS7GDPVIH57SQ, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.815007834 (540h52m18.686090879s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:51332->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:21:18.815017242 (540h52m18.686100287s) [noise=true , address=45.91.116.12:4225 , piece_id=GQZXTYRLJI4Y7P6R4SF7MUZ6Y6IWEO5U5EJGDAUFPSDCLCXHCVNA, seg_pos={0 0}, upload=474335]: Long tail cancellation due to optimal threshold Dec 28 04:21:18.815082696 (540h52m18.686165732s) [seg_pos={0 0}, upload=474335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:21:18.815164993 (540h52m18.686248028s) [upload=474335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:21:18.815169401 (540h52m18.686252436s) [upload=474335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:21:19.112094479 (540h52m18.983177524s) [upload=474335]: Done uploading segments... Dec 28 04:21:19.112116741 (540h52m18.983199776s) [upload=474335]: Upload finished. err: <nil> Duration: 30.004802745s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X......X...........] duration: 41.199609596s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:21:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 41.199635695s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X.X...X..X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX..] duration: 3.863886842s trace: https://jaeger.dev.storj.io/trace/36c67681f69a29b0 FAIL (2025-12-28 04:21:11 UTC) orbiter warning: context deadline exceeded Duration: 3.863913311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX...] duration: 7.287945674s trace: https://jaeger.dev.storj.io/trace/52794ad607073d7c FAIL (2025-12-28 04:21:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :18:47.073237416 (540h49m46.944320451s) [upload=474300]: Starting upload Dec 28 04:18:47.073287732 (540h49m46.944370777s) [upload=474300]: Done starting upload Dec 28 04:18:47.073349640 (540h49m46.944432675s) [upload=474300]: Uploading segments... Dec 28 04:18:47.073460260 (540h49m46.944543295s) [upload=474300]: Got next segment. Inline: false Dec 28 04:18:47.073486270 (540h49m46.944569305s) [upload=474300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:18:47.073490157 (540h49m46.944573192s) [upload=474300]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:18:47.317713397 (540h49m47.188796442s) [seg_pos={0 0}, upload=474300]: Begin upload segment... Dec 28 04:18:47.317771858 (540h49m47.188854903s) [seg_pos={0 0}, upload=474300]: Done begin upload segment. Dec 28 04:18:47.317860316 (540h49m47.188943352s) [noise=true , address=145.79.227.7:4293 , piece_id=45JGMMBUFM22VKZCKYPNZHHDYTXNXNBFOVMETABAY6XKCXKCIRUA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.317938425 (540h49m47.18902146s) [noise=true , address=145.79.227.39:4013 , piece_id=3XU5U3GDHXTRAEHN5PUWLEKGGQAGO5ASDKG3QHAGBS2KJUPWDURQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.318529030 (540h49m47.189612075s) [noise=true , address=145.79.227.33:4869 , piece_id=B2F6MM46WWJIXHVKZAJC5FQ27FQS73X2MUDFYVSTQJZNBHIN4EBA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.319015466 (540h49m47.190098501s) [noise=true , address=77.90.31.19:4841 , piece_id=KSQ7B5DSFPBS5RFWUKSCWRK6L7DCZJS5HUMIMZEGB6CGVHTVUKWQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.319439854 (540h49m47.190522899s) [noise=true , address=145.79.227.50:4375 , piece_id=HGWOOIH7EYEZGIVBUO3B5AERNA4NQFEQZD4OQTY4N3QA5TL2AO5Q, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.319815359 (540h49m47.190898404s) [noise=true , address=45.91.116.7:4012 , piece_id=7AL3GSECL5YG63KPU3GIN5VQK7Y463WAWTDB22RXUKATRSYCQHVA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.320232834 (540h49m47.191315869s) [noise=true , address=45.91.116.10:4084 , piece_id=P7AZAWCXJDXCKVJ7V7SGSZHF5VEAGMOEOTKNV5M62QOU4HL2ALRA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.320619951 (540h49m47.191702996s) [noise=true , address=77.90.31.18:4805 , piece_id=5CAGZW6RQYUUM76YBS5GWB76D7HPIN7B5M5NZBZLXJD2P5FL2A3Q, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.321095406 (540h49m47.192178451s) [noise=true , address=145.79.227.32:4835 , piece_id=I4NLLATABXJIEA2VE3272C6VS5NYEVHRGIKFDV2O7MPFII2MTCBA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.323109261 (540h49m47.194192296s) [noise=true , address=45.91.116.12:4225 , piece_id=B7PFPJM6JJ5K2EIR7ALTVWUHQF7TOSFOFDMGDO23LVRIJASTPPNQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.323529391 (540h49m47.194612436s) [noise=true , address=145.79.227.44:4195 , piece_id=4C6D5O454BO27S2MO5WZN656QTDVYOIY5CSMC7JGLU2YCUL3KCIA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.325116903 (540h49m47.196199948s) [noise=true , address=145.79.227.36:4519 , piece_id=SJH6RTQRTNNHR5QRTDHWZWMOEMMSUNTBPZQC5PCVRSTH4MVLBQLQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.326455813 (540h49m47.197538858s) [noise=true , address=145.79.227.13:4804 , piece_id=6KU5HJJMJ5V2MDZQTHXZVIH2OCFGYWP4YJKWS52NXYNC3VTUNVQQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.328790888 (540h49m47.199873923s) [noise=true , address=145.79.227.6:4258 , piece_id=DWNYQ4YEHUN4OIYAXI3WLLZ6SN2AURVNAPJQMZ7UVJVRRVLNRUOA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.330709962 (540h49m47.201792997s) [noise=true , address=77.90.31.11:4556 , piece_id=IB3P7XN5KJW3JYBSA7DJMAPQC73RWUMQFJ7X3RBT4G4BYO3OPGGA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.332439606 (540h49m47.203522651s) [noise=true , address=77.90.31.10:4519 , piece_id=R2HCMXCQ4ETJWJZDZ3MQW73OZNU4FJLOVBW3FI6FSJF3YL2RHUUQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:49.035275971 (540h49m48.906359016s) [upload=474300]: Next returned nil segment Dec 28 04:18:49.035293936 (540h49m48.906376971s) [upload=474300]: Waiting for error group managing segments... Dec 28 04:18:49.319051130 (540h49m49.190134165s) [noise=true , address=145.79.227.32:4835 , piece_id=I4NLLATABXJIEA2VE3272C6VS5NYEVHRGIKFDV2O7MPFII2MTCBA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.328655147 (540h49m49.199738192s) [noise=true , address=145.79.227.50:4375 , piece_id=HGWOOIH7EYEZGIVBUO3B5AERNA4NQFEQZD4OQTY4N3QA5TL2AO5Q, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.345084158 (540h49m49.216167214s) [noise=true , address=145.79.227.36:4519 , piece_id=SJH6RTQRTNNHR5QRTDHWZWMOEMMSUNTBPZQC5PCVRSTH4MVLBQLQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.357761868 (540h49m49.228844913s) [noise=true , address=145.79.227.7:4293 , piece_id=45JGMMBUFM22VKZCKYPNZHHDYTXNXNBFOVMETABAY6XKCXKCIRUA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.361729221 (540h49m49.232812256s) [noise=true , address=145.79.227.13:4804 , piece_id=6KU5HJJMJ5V2MDZQTHXZVIH2OCFGYWP4YJKWS52NXYNC3VTUNVQQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.368272281 (540h49m49.239355326s) [noise=true , address=145.79.227.6:4258 , piece_id=DWNYQ4YEHUN4OIYAXI3WLLZ6SN2AURVNAPJQMZ7UVJVRRVLNRUOA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.525625362 (540h49m49.396708397s) [noise=true , address=77.90.31.11:4556 , piece_id=IB3P7XN5KJW3JYBSA7DJMAPQC73RWUMQFJ7X3RBT4G4BYO3OPGGA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.555733290 (540h49m49.426816325s) [noise=true , address=77.90.31.10:4519 , piece_id=R2HCMXCQ4ETJWJZDZ3MQW73OZNU4FJLOVBW3FI6FSJF3YL2RHUUQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.616709523 (540h49m49.487792568s) [noise=true , address=45.91.116.7:4012 , piece_id=7AL3GSECL5YG63KPU3GIN5VQK7Y463WAWTDB22RXUKATRSYCQHVA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.626950633 (540h49m49.498033668s) [noise=true , address=77.90.31.18:4805 , piece_id=5CAGZW6RQYUUM76YBS5GWB76D7HPIN7B5M5NZBZLXJD2P5FL2A3Q, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.782757851 (540h49m49.653840886s) [noise=true , address=145.79.227.39:4013 , piece_id=3XU5U3GDHXTRAEHN5PUWLEKGGQAGO5ASDKG3QHAGBS2KJUPWDURQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.847377061 (540h49m49.718460096s) [noise=true , address=77.90.31.19:4841 , piece_id=KSQ7B5DSFPBS5RFWUKSCWRK6L7DCZJS5HUMIMZEGB6CGVHTVUKWQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:50.037733874 (540h49m49.908816909s) [noise=true , address=45.91.116.12:4225 , piece_id=B7PFPJM6JJ5K2EIR7ALTVWUHQF7TOSFOFDMGDO23LVRIJASTPPNQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:50.037752860 (540h49m49.908835905s) [seg_pos={0 0}, upload=474300]: Segment reached optimal threshold of 13 pieces. Dec 28 04:18:50.038011783 (540h49m49.909094828s) [noise=true , address=45.91.116.10:4084 , piece_id=P7AZAWCXJDXCKVJ7V7SGSZHF5VEAGMOEOTKNV5M62QOU4HL2ALRA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:59984->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:59984->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59984->45.91.116.10:4084: use of closed network connection Dec 28 04:18:50.038019138 (540h49m49.909102183s) [noise=true , address=45.91.116.10:4084 , piece_id=P7AZAWCXJDXCKVJ7V7SGSZHF5VEAGMOEOTKNV5M62QOU4HL2ALRA, seg_pos={0 0}, upload=474300]: Long tail cancellation due to optimal threshold Dec 28 04:18:50.038312466 (540h49m49.909395501s) [noise=true , address=145.79.227.33:4869 , piece_id=B2F6MM46WWJIXHVKZAJC5FQ27FQS73X2MUDFYVSTQJZNBHIN4EBA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 04:18:50.038319900 (540h49m49.909402945s) [noise=true , address=145.79.227.33:4869 , piece_id=B2F6MM46WWJIXHVKZAJC5FQ27FQS73X2MUDFYVSTQJZNBHIN4EBA, seg_pos={0 0}, upload=474300]: Long tail cancellation due to optimal threshold Dec 28 04:18:50.038450769 (540h49m49.909533804s) [noise=true , address=145.79.227.44:4195 , piece_id=4C6D5O454BO27S2MO5WZN656QTDVYOIY5CSMC7JGLU2YCUL3KCIA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:18:50.038458353 (540h49m49.909541398s) [noise=true , address=145.79.227.44:4195 , piece_id=4C6D5O454BO27S2MO5WZN656QTDVYOIY5CSMC7JGLU2YCUL3KCIA, seg_pos={0 0}, upload=474300]: Long tail cancellation due to optimal threshold Dec 28 04:18:50.038479664 (540h49m49.909562699s) [seg_pos={0 0}, upload=474300]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:18:50.038565016 (540h49m49.909648051s) [upload=474300]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:18:50.038570337 (540h49m49.909653372s) [upload=474300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:18:50.328489033 (540h49m50.199572088s) [upload=474300]: Done uploading segments... Dec 28 04:18:50.328513149 (540h49m50.199596185s) [upload=474300]: Upload finished. err: <nil> Duration: 7.287978346s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X.X.X...X..X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX.] duration: 3.714410841s trace: https://jaeger.dev.storj.io/trace/36c67681f69a29b0 FAIL (2025-12-28 04:20:55 UTC) orbiter warning: context deadline exceeded Duration: 3.714501733s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 9.284415966s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:20:44 UTC) orbiter warning: context deadline exceeded Duration: 9.284503613s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX..] duration: 20.932831463s trace: https://jaeger.dev.storj.io/trace/52794ad607073d7c FAIL (2025-12-28 04:20:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :18:47.073237416 (540h49m46.944320451s) [upload=474300]: Starting upload Dec 28 04:18:47.073287732 (540h49m46.944370777s) [upload=474300]: Done starting upload Dec 28 04:18:47.073349640 (540h49m46.944432675s) [upload=474300]: Uploading segments... Dec 28 04:18:47.073460260 (540h49m46.944543295s) [upload=474300]: Got next segment. Inline: false Dec 28 04:18:47.073486270 (540h49m46.944569305s) [upload=474300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:18:47.073490157 (540h49m46.944573192s) [upload=474300]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:18:47.317713397 (540h49m47.188796442s) [seg_pos={0 0}, upload=474300]: Begin upload segment... Dec 28 04:18:47.317771858 (540h49m47.188854903s) [seg_pos={0 0}, upload=474300]: Done begin upload segment. Dec 28 04:18:47.317860316 (540h49m47.188943352s) [noise=true , address=145.79.227.7:4293 , piece_id=45JGMMBUFM22VKZCKYPNZHHDYTXNXNBFOVMETABAY6XKCXKCIRUA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.317938425 (540h49m47.18902146s) [noise=true , address=145.79.227.39:4013 , piece_id=3XU5U3GDHXTRAEHN5PUWLEKGGQAGO5ASDKG3QHAGBS2KJUPWDURQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.318529030 (540h49m47.189612075s) [noise=true , address=145.79.227.33:4869 , piece_id=B2F6MM46WWJIXHVKZAJC5FQ27FQS73X2MUDFYVSTQJZNBHIN4EBA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.319015466 (540h49m47.190098501s) [noise=true , address=77.90.31.19:4841 , piece_id=KSQ7B5DSFPBS5RFWUKSCWRK6L7DCZJS5HUMIMZEGB6CGVHTVUKWQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.319439854 (540h49m47.190522899s) [noise=true , address=145.79.227.50:4375 , piece_id=HGWOOIH7EYEZGIVBUO3B5AERNA4NQFEQZD4OQTY4N3QA5TL2AO5Q, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.319815359 (540h49m47.190898404s) [noise=true , address=45.91.116.7:4012 , piece_id=7AL3GSECL5YG63KPU3GIN5VQK7Y463WAWTDB22RXUKATRSYCQHVA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.320232834 (540h49m47.191315869s) [noise=true , address=45.91.116.10:4084 , piece_id=P7AZAWCXJDXCKVJ7V7SGSZHF5VEAGMOEOTKNV5M62QOU4HL2ALRA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.320619951 (540h49m47.191702996s) [noise=true , address=77.90.31.18:4805 , piece_id=5CAGZW6RQYUUM76YBS5GWB76D7HPIN7B5M5NZBZLXJD2P5FL2A3Q, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.321095406 (540h49m47.192178451s) [noise=true , address=145.79.227.32:4835 , piece_id=I4NLLATABXJIEA2VE3272C6VS5NYEVHRGIKFDV2O7MPFII2MTCBA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.323109261 (540h49m47.194192296s) [noise=true , address=45.91.116.12:4225 , piece_id=B7PFPJM6JJ5K2EIR7ALTVWUHQF7TOSFOFDMGDO23LVRIJASTPPNQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.323529391 (540h49m47.194612436s) [noise=true , address=145.79.227.44:4195 , piece_id=4C6D5O454BO27S2MO5WZN656QTDVYOIY5CSMC7JGLU2YCUL3KCIA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.325116903 (540h49m47.196199948s) [noise=true , address=145.79.227.36:4519 , piece_id=SJH6RTQRTNNHR5QRTDHWZWMOEMMSUNTBPZQC5PCVRSTH4MVLBQLQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.326455813 (540h49m47.197538858s) [noise=true , address=145.79.227.13:4804 , piece_id=6KU5HJJMJ5V2MDZQTHXZVIH2OCFGYWP4YJKWS52NXYNC3VTUNVQQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.328790888 (540h49m47.199873923s) [noise=true , address=145.79.227.6:4258 , piece_id=DWNYQ4YEHUN4OIYAXI3WLLZ6SN2AURVNAPJQMZ7UVJVRRVLNRUOA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.330709962 (540h49m47.201792997s) [noise=true , address=77.90.31.11:4556 , piece_id=IB3P7XN5KJW3JYBSA7DJMAPQC73RWUMQFJ7X3RBT4G4BYO3OPGGA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.332439606 (540h49m47.203522651s) [noise=true , address=77.90.31.10:4519 , piece_id=R2HCMXCQ4ETJWJZDZ3MQW73OZNU4FJLOVBW3FI6FSJF3YL2RHUUQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:49.035275971 (540h49m48.906359016s) [upload=474300]: Next returned nil segment Dec 28 04:18:49.035293936 (540h49m48.906376971s) [upload=474300]: Waiting for error group managing segments... Dec 28 04:18:49.319051130 (540h49m49.190134165s) [noise=true , address=145.79.227.32:4835 , piece_id=I4NLLATABXJIEA2VE3272C6VS5NYEVHRGIKFDV2O7MPFII2MTCBA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.328655147 (540h49m49.199738192s) [noise=true , address=145.79.227.50:4375 , piece_id=HGWOOIH7EYEZGIVBUO3B5AERNA4NQFEQZD4OQTY4N3QA5TL2AO5Q, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.345084158 (540h49m49.216167214s) [noise=true , address=145.79.227.36:4519 , piece_id=SJH6RTQRTNNHR5QRTDHWZWMOEMMSUNTBPZQC5PCVRSTH4MVLBQLQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.357761868 (540h49m49.228844913s) [noise=true , address=145.79.227.7:4293 , piece_id=45JGMMBUFM22VKZCKYPNZHHDYTXNXNBFOVMETABAY6XKCXKCIRUA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.361729221 (540h49m49.232812256s) [noise=true , address=145.79.227.13:4804 , piece_id=6KU5HJJMJ5V2MDZQTHXZVIH2OCFGYWP4YJKWS52NXYNC3VTUNVQQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.368272281 (540h49m49.239355326s) [noise=true , address=145.79.227.6:4258 , piece_id=DWNYQ4YEHUN4OIYAXI3WLLZ6SN2AURVNAPJQMZ7UVJVRRVLNRUOA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.525625362 (540h49m49.396708397s) [noise=true , address=77.90.31.11:4556 , piece_id=IB3P7XN5KJW3JYBSA7DJMAPQC73RWUMQFJ7X3RBT4G4BYO3OPGGA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.555733290 (540h49m49.426816325s) [noise=true , address=77.90.31.10:4519 , piece_id=R2HCMXCQ4ETJWJZDZ3MQW73OZNU4FJLOVBW3FI6FSJF3YL2RHUUQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.616709523 (540h49m49.487792568s) [noise=true , address=45.91.116.7:4012 , piece_id=7AL3GSECL5YG63KPU3GIN5VQK7Y463WAWTDB22RXUKATRSYCQHVA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.626950633 (540h49m49.498033668s) [noise=true , address=77.90.31.18:4805 , piece_id=5CAGZW6RQYUUM76YBS5GWB76D7HPIN7B5M5NZBZLXJD2P5FL2A3Q, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.782757851 (540h49m49.653840886s) [noise=true , address=145.79.227.39:4013 , piece_id=3XU5U3GDHXTRAEHN5PUWLEKGGQAGO5ASDKG3QHAGBS2KJUPWDURQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.847377061 (540h49m49.718460096s) [noise=true , address=77.90.31.19:4841 , piece_id=KSQ7B5DSFPBS5RFWUKSCWRK6L7DCZJS5HUMIMZEGB6CGVHTVUKWQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:50.037733874 (540h49m49.908816909s) [noise=true , address=45.91.116.12:4225 , piece_id=B7PFPJM6JJ5K2EIR7ALTVWUHQF7TOSFOFDMGDO23LVRIJASTPPNQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:50.037752860 (540h49m49.908835905s) [seg_pos={0 0}, upload=474300]: Segment reached optimal threshold of 13 pieces. Dec 28 04:18:50.038011783 (540h49m49.909094828s) [noise=true , address=45.91.116.10:4084 , piece_id=P7AZAWCXJDXCKVJ7V7SGSZHF5VEAGMOEOTKNV5M62QOU4HL2ALRA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:59984->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:59984->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59984->45.91.116.10:4084: use of closed network connection Dec 28 04:18:50.038019138 (540h49m49.909102183s) [noise=true , address=45.91.116.10:4084 , piece_id=P7AZAWCXJDXCKVJ7V7SGSZHF5VEAGMOEOTKNV5M62QOU4HL2ALRA, seg_pos={0 0}, upload=474300]: Long tail cancellation due to optimal threshold Dec 28 04:18:50.038312466 (540h49m49.909395501s) [noise=true , address=145.79.227.33:4869 , piece_id=B2F6MM46WWJIXHVKZAJC5FQ27FQS73X2MUDFYVSTQJZNBHIN4EBA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 04:18:50.038319900 (540h49m49.909402945s) [noise=true , address=145.79.227.33:4869 , piece_id=B2F6MM46WWJIXHVKZAJC5FQ27FQS73X2MUDFYVSTQJZNBHIN4EBA, seg_pos={0 0}, upload=474300]: Long tail cancellation due to optimal threshold Dec 28 04:18:50.038450769 (540h49m49.909533804s) [noise=true , address=145.79.227.44:4195 , piece_id=4C6D5O454BO27S2MO5WZN656QTDVYOIY5CSMC7JGLU2YCUL3KCIA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:18:50.038458353 (540h49m49.909541398s) [noise=true , address=145.79.227.44:4195 , piece_id=4C6D5O454BO27S2MO5WZN656QTDVYOIY5CSMC7JGLU2YCUL3KCIA, seg_pos={0 0}, upload=474300]: Long tail cancellation due to optimal threshold Dec 28 04:18:50.038479664 (540h49m49.909562699s) [seg_pos={0 0}, upload=474300]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:18:50.038565016 (540h49m49.909648051s) [upload=474300]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:18:50.038570337 (540h49m49.909653372s) [upload=474300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:18:50.328489033 (540h49m50.199572088s) [upload=474300]: Done uploading segments... Dec 28 04:18:50.328513149 (540h49m50.199596185s) [upload=474300]: Upload finished. err: <nil> Duration: 20.932854387s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X......X..........] duration: 38.841124349s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:20:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 38.841141823s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X......X.........] duration: 32.453502633s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:19:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 32.453523983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.X.X.X...X..X.X.X..X...XX.....X......XX...X.X....X.X...XXXXX] duration: 5.001658116s trace: https://jaeger.dev.storj.io/trace/36c67681f69a29b0 FAIL (2025-12-28 04:19:52 UTC) orbiter warning: context deadline exceeded Duration: 5.001674568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX.] duration: 6.633736593s trace: https://jaeger.dev.storj.io/trace/52794ad607073d7c FAIL (2025-12-28 04:19:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :18:47.073237416 (540h49m46.944320451s) [upload=474300]: Starting upload Dec 28 04:18:47.073287732 (540h49m46.944370777s) [upload=474300]: Done starting upload Dec 28 04:18:47.073349640 (540h49m46.944432675s) [upload=474300]: Uploading segments... Dec 28 04:18:47.073460260 (540h49m46.944543295s) [upload=474300]: Got next segment. Inline: false Dec 28 04:18:47.073486270 (540h49m46.944569305s) [upload=474300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:18:47.073490157 (540h49m46.944573192s) [upload=474300]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:18:47.317713397 (540h49m47.188796442s) [seg_pos={0 0}, upload=474300]: Begin upload segment... Dec 28 04:18:47.317771858 (540h49m47.188854903s) [seg_pos={0 0}, upload=474300]: Done begin upload segment. Dec 28 04:18:47.317860316 (540h49m47.188943352s) [noise=true , address=145.79.227.7:4293 , piece_id=45JGMMBUFM22VKZCKYPNZHHDYTXNXNBFOVMETABAY6XKCXKCIRUA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.317938425 (540h49m47.18902146s) [noise=true , address=145.79.227.39:4013 , piece_id=3XU5U3GDHXTRAEHN5PUWLEKGGQAGO5ASDKG3QHAGBS2KJUPWDURQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.318529030 (540h49m47.189612075s) [noise=true , address=145.79.227.33:4869 , piece_id=B2F6MM46WWJIXHVKZAJC5FQ27FQS73X2MUDFYVSTQJZNBHIN4EBA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.319015466 (540h49m47.190098501s) [noise=true , address=77.90.31.19:4841 , piece_id=KSQ7B5DSFPBS5RFWUKSCWRK6L7DCZJS5HUMIMZEGB6CGVHTVUKWQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.319439854 (540h49m47.190522899s) [noise=true , address=145.79.227.50:4375 , piece_id=HGWOOIH7EYEZGIVBUO3B5AERNA4NQFEQZD4OQTY4N3QA5TL2AO5Q, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.319815359 (540h49m47.190898404s) [noise=true , address=45.91.116.7:4012 , piece_id=7AL3GSECL5YG63KPU3GIN5VQK7Y463WAWTDB22RXUKATRSYCQHVA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.320232834 (540h49m47.191315869s) [noise=true , address=45.91.116.10:4084 , piece_id=P7AZAWCXJDXCKVJ7V7SGSZHF5VEAGMOEOTKNV5M62QOU4HL2ALRA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.320619951 (540h49m47.191702996s) [noise=true , address=77.90.31.18:4805 , piece_id=5CAGZW6RQYUUM76YBS5GWB76D7HPIN7B5M5NZBZLXJD2P5FL2A3Q, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.321095406 (540h49m47.192178451s) [noise=true , address=145.79.227.32:4835 , piece_id=I4NLLATABXJIEA2VE3272C6VS5NYEVHRGIKFDV2O7MPFII2MTCBA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.323109261 (540h49m47.194192296s) [noise=true , address=45.91.116.12:4225 , piece_id=B7PFPJM6JJ5K2EIR7ALTVWUHQF7TOSFOFDMGDO23LVRIJASTPPNQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.323529391 (540h49m47.194612436s) [noise=true , address=145.79.227.44:4195 , piece_id=4C6D5O454BO27S2MO5WZN656QTDVYOIY5CSMC7JGLU2YCUL3KCIA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.325116903 (540h49m47.196199948s) [noise=true , address=145.79.227.36:4519 , piece_id=SJH6RTQRTNNHR5QRTDHWZWMOEMMSUNTBPZQC5PCVRSTH4MVLBQLQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.326455813 (540h49m47.197538858s) [noise=true , address=145.79.227.13:4804 , piece_id=6KU5HJJMJ5V2MDZQTHXZVIH2OCFGYWP4YJKWS52NXYNC3VTUNVQQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.328790888 (540h49m47.199873923s) [noise=true , address=145.79.227.6:4258 , piece_id=DWNYQ4YEHUN4OIYAXI3WLLZ6SN2AURVNAPJQMZ7UVJVRRVLNRUOA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.330709962 (540h49m47.201792997s) [noise=true , address=77.90.31.11:4556 , piece_id=IB3P7XN5KJW3JYBSA7DJMAPQC73RWUMQFJ7X3RBT4G4BYO3OPGGA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.332439606 (540h49m47.203522651s) [noise=true , address=77.90.31.10:4519 , piece_id=R2HCMXCQ4ETJWJZDZ3MQW73OZNU4FJLOVBW3FI6FSJF3YL2RHUUQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:49.035275971 (540h49m48.906359016s) [upload=474300]: Next returned nil segment Dec 28 04:18:49.035293936 (540h49m48.906376971s) [upload=474300]: Waiting for error group managing segments... Dec 28 04:18:49.319051130 (540h49m49.190134165s) [noise=true , address=145.79.227.32:4835 , piece_id=I4NLLATABXJIEA2VE3272C6VS5NYEVHRGIKFDV2O7MPFII2MTCBA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.328655147 (540h49m49.199738192s) [noise=true , address=145.79.227.50:4375 , piece_id=HGWOOIH7EYEZGIVBUO3B5AERNA4NQFEQZD4OQTY4N3QA5TL2AO5Q, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.345084158 (540h49m49.216167214s) [noise=true , address=145.79.227.36:4519 , piece_id=SJH6RTQRTNNHR5QRTDHWZWMOEMMSUNTBPZQC5PCVRSTH4MVLBQLQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.357761868 (540h49m49.228844913s) [noise=true , address=145.79.227.7:4293 , piece_id=45JGMMBUFM22VKZCKYPNZHHDYTXNXNBFOVMETABAY6XKCXKCIRUA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.361729221 (540h49m49.232812256s) [noise=true , address=145.79.227.13:4804 , piece_id=6KU5HJJMJ5V2MDZQTHXZVIH2OCFGYWP4YJKWS52NXYNC3VTUNVQQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.368272281 (540h49m49.239355326s) [noise=true , address=145.79.227.6:4258 , piece_id=DWNYQ4YEHUN4OIYAXI3WLLZ6SN2AURVNAPJQMZ7UVJVRRVLNRUOA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.525625362 (540h49m49.396708397s) [noise=true , address=77.90.31.11:4556 , piece_id=IB3P7XN5KJW3JYBSA7DJMAPQC73RWUMQFJ7X3RBT4G4BYO3OPGGA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.555733290 (540h49m49.426816325s) [noise=true , address=77.90.31.10:4519 , piece_id=R2HCMXCQ4ETJWJZDZ3MQW73OZNU4FJLOVBW3FI6FSJF3YL2RHUUQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.616709523 (540h49m49.487792568s) [noise=true , address=45.91.116.7:4012 , piece_id=7AL3GSECL5YG63KPU3GIN5VQK7Y463WAWTDB22RXUKATRSYCQHVA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.626950633 (540h49m49.498033668s) [noise=true , address=77.90.31.18:4805 , piece_id=5CAGZW6RQYUUM76YBS5GWB76D7HPIN7B5M5NZBZLXJD2P5FL2A3Q, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.782757851 (540h49m49.653840886s) [noise=true , address=145.79.227.39:4013 , piece_id=3XU5U3GDHXTRAEHN5PUWLEKGGQAGO5ASDKG3QHAGBS2KJUPWDURQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.847377061 (540h49m49.718460096s) [noise=true , address=77.90.31.19:4841 , piece_id=KSQ7B5DSFPBS5RFWUKSCWRK6L7DCZJS5HUMIMZEGB6CGVHTVUKWQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:50.037733874 (540h49m49.908816909s) [noise=true , address=45.91.116.12:4225 , piece_id=B7PFPJM6JJ5K2EIR7ALTVWUHQF7TOSFOFDMGDO23LVRIJASTPPNQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:50.037752860 (540h49m49.908835905s) [seg_pos={0 0}, upload=474300]: Segment reached optimal threshold of 13 pieces. Dec 28 04:18:50.038011783 (540h49m49.909094828s) [noise=true , address=45.91.116.10:4084 , piece_id=P7AZAWCXJDXCKVJ7V7SGSZHF5VEAGMOEOTKNV5M62QOU4HL2ALRA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:59984->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:59984->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59984->45.91.116.10:4084: use of closed network connection Dec 28 04:18:50.038019138 (540h49m49.909102183s) [noise=true , address=45.91.116.10:4084 , piece_id=P7AZAWCXJDXCKVJ7V7SGSZHF5VEAGMOEOTKNV5M62QOU4HL2ALRA, seg_pos={0 0}, upload=474300]: Long tail cancellation due to optimal threshold Dec 28 04:18:50.038312466 (540h49m49.909395501s) [noise=true , address=145.79.227.33:4869 , piece_id=B2F6MM46WWJIXHVKZAJC5FQ27FQS73X2MUDFYVSTQJZNBHIN4EBA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 04:18:50.038319900 (540h49m49.909402945s) [noise=true , address=145.79.227.33:4869 , piece_id=B2F6MM46WWJIXHVKZAJC5FQ27FQS73X2MUDFYVSTQJZNBHIN4EBA, seg_pos={0 0}, upload=474300]: Long tail cancellation due to optimal threshold Dec 28 04:18:50.038450769 (540h49m49.909533804s) [noise=true , address=145.79.227.44:4195 , piece_id=4C6D5O454BO27S2MO5WZN656QTDVYOIY5CSMC7JGLU2YCUL3KCIA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:18:50.038458353 (540h49m49.909541398s) [noise=true , address=145.79.227.44:4195 , piece_id=4C6D5O454BO27S2MO5WZN656QTDVYOIY5CSMC7JGLU2YCUL3KCIA, seg_pos={0 0}, upload=474300]: Long tail cancellation due to optimal threshold Dec 28 04:18:50.038479664 (540h49m49.909562699s) [seg_pos={0 0}, upload=474300]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:18:50.038565016 (540h49m49.909648051s) [upload=474300]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:18:50.038570337 (540h49m49.909653372s) [upload=474300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:18:50.328489033 (540h49m50.199572088s) [upload=474300]: Done uploading segments... Dec 28 04:18:50.328513149 (540h49m50.199596185s) [upload=474300]: Upload finished. err: <nil> Duration: 6.633761731s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.733013146s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:19:28 UTC) orbiter warning: context deadline exceeded Duration: 6.73304696s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX..XX] duration: 30.002988928s trace: https://jaeger.dev.storj.io/trace/52794ad607073d7c FAIL (2025-12-28 04:19:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :18:47.073237416 (540h49m46.944320451s) [upload=474300]: Starting upload Dec 28 04:18:47.073287732 (540h49m46.944370777s) [upload=474300]: Done starting upload Dec 28 04:18:47.073349640 (540h49m46.944432675s) [upload=474300]: Uploading segments... Dec 28 04:18:47.073460260 (540h49m46.944543295s) [upload=474300]: Got next segment. Inline: false Dec 28 04:18:47.073486270 (540h49m46.944569305s) [upload=474300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:18:47.073490157 (540h49m46.944573192s) [upload=474300]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:18:47.317713397 (540h49m47.188796442s) [seg_pos={0 0}, upload=474300]: Begin upload segment... Dec 28 04:18:47.317771858 (540h49m47.188854903s) [seg_pos={0 0}, upload=474300]: Done begin upload segment. Dec 28 04:18:47.317860316 (540h49m47.188943352s) [noise=true , address=145.79.227.7:4293 , piece_id=45JGMMBUFM22VKZCKYPNZHHDYTXNXNBFOVMETABAY6XKCXKCIRUA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.317938425 (540h49m47.18902146s) [noise=true , address=145.79.227.39:4013 , piece_id=3XU5U3GDHXTRAEHN5PUWLEKGGQAGO5ASDKG3QHAGBS2KJUPWDURQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.318529030 (540h49m47.189612075s) [noise=true , address=145.79.227.33:4869 , piece_id=B2F6MM46WWJIXHVKZAJC5FQ27FQS73X2MUDFYVSTQJZNBHIN4EBA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.319015466 (540h49m47.190098501s) [noise=true , address=77.90.31.19:4841 , piece_id=KSQ7B5DSFPBS5RFWUKSCWRK6L7DCZJS5HUMIMZEGB6CGVHTVUKWQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.319439854 (540h49m47.190522899s) [noise=true , address=145.79.227.50:4375 , piece_id=HGWOOIH7EYEZGIVBUO3B5AERNA4NQFEQZD4OQTY4N3QA5TL2AO5Q, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.319815359 (540h49m47.190898404s) [noise=true , address=45.91.116.7:4012 , piece_id=7AL3GSECL5YG63KPU3GIN5VQK7Y463WAWTDB22RXUKATRSYCQHVA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.320232834 (540h49m47.191315869s) [noise=true , address=45.91.116.10:4084 , piece_id=P7AZAWCXJDXCKVJ7V7SGSZHF5VEAGMOEOTKNV5M62QOU4HL2ALRA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.320619951 (540h49m47.191702996s) [noise=true , address=77.90.31.18:4805 , piece_id=5CAGZW6RQYUUM76YBS5GWB76D7HPIN7B5M5NZBZLXJD2P5FL2A3Q, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.321095406 (540h49m47.192178451s) [noise=true , address=145.79.227.32:4835 , piece_id=I4NLLATABXJIEA2VE3272C6VS5NYEVHRGIKFDV2O7MPFII2MTCBA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.323109261 (540h49m47.194192296s) [noise=true , address=45.91.116.12:4225 , piece_id=B7PFPJM6JJ5K2EIR7ALTVWUHQF7TOSFOFDMGDO23LVRIJASTPPNQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.323529391 (540h49m47.194612436s) [noise=true , address=145.79.227.44:4195 , piece_id=4C6D5O454BO27S2MO5WZN656QTDVYOIY5CSMC7JGLU2YCUL3KCIA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.325116903 (540h49m47.196199948s) [noise=true , address=145.79.227.36:4519 , piece_id=SJH6RTQRTNNHR5QRTDHWZWMOEMMSUNTBPZQC5PCVRSTH4MVLBQLQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.326455813 (540h49m47.197538858s) [noise=true , address=145.79.227.13:4804 , piece_id=6KU5HJJMJ5V2MDZQTHXZVIH2OCFGYWP4YJKWS52NXYNC3VTUNVQQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.328790888 (540h49m47.199873923s) [noise=true , address=145.79.227.6:4258 , piece_id=DWNYQ4YEHUN4OIYAXI3WLLZ6SN2AURVNAPJQMZ7UVJVRRVLNRUOA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.330709962 (540h49m47.201792997s) [noise=true , address=77.90.31.11:4556 , piece_id=IB3P7XN5KJW3JYBSA7DJMAPQC73RWUMQFJ7X3RBT4G4BYO3OPGGA, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:47.332439606 (540h49m47.203522651s) [noise=true , address=77.90.31.10:4519 , piece_id=R2HCMXCQ4ETJWJZDZ3MQW73OZNU4FJLOVBW3FI6FSJF3YL2RHUUQ, seg_pos={0 0}, upload=474300]: Uploading piece... Dec 28 04:18:49.035275971 (540h49m48.906359016s) [upload=474300]: Next returned nil segment Dec 28 04:18:49.035293936 (540h49m48.906376971s) [upload=474300]: Waiting for error group managing segments... Dec 28 04:18:49.319051130 (540h49m49.190134165s) [noise=true , address=145.79.227.32:4835 , piece_id=I4NLLATABXJIEA2VE3272C6VS5NYEVHRGIKFDV2O7MPFII2MTCBA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.328655147 (540h49m49.199738192s) [noise=true , address=145.79.227.50:4375 , piece_id=HGWOOIH7EYEZGIVBUO3B5AERNA4NQFEQZD4OQTY4N3QA5TL2AO5Q, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.345084158 (540h49m49.216167214s) [noise=true , address=145.79.227.36:4519 , piece_id=SJH6RTQRTNNHR5QRTDHWZWMOEMMSUNTBPZQC5PCVRSTH4MVLBQLQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.357761868 (540h49m49.228844913s) [noise=true , address=145.79.227.7:4293 , piece_id=45JGMMBUFM22VKZCKYPNZHHDYTXNXNBFOVMETABAY6XKCXKCIRUA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.361729221 (540h49m49.232812256s) [noise=true , address=145.79.227.13:4804 , piece_id=6KU5HJJMJ5V2MDZQTHXZVIH2OCFGYWP4YJKWS52NXYNC3VTUNVQQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.368272281 (540h49m49.239355326s) [noise=true , address=145.79.227.6:4258 , piece_id=DWNYQ4YEHUN4OIYAXI3WLLZ6SN2AURVNAPJQMZ7UVJVRRVLNRUOA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.525625362 (540h49m49.396708397s) [noise=true , address=77.90.31.11:4556 , piece_id=IB3P7XN5KJW3JYBSA7DJMAPQC73RWUMQFJ7X3RBT4G4BYO3OPGGA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.555733290 (540h49m49.426816325s) [noise=true , address=77.90.31.10:4519 , piece_id=R2HCMXCQ4ETJWJZDZ3MQW73OZNU4FJLOVBW3FI6FSJF3YL2RHUUQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.616709523 (540h49m49.487792568s) [noise=true , address=45.91.116.7:4012 , piece_id=7AL3GSECL5YG63KPU3GIN5VQK7Y463WAWTDB22RXUKATRSYCQHVA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.626950633 (540h49m49.498033668s) [noise=true , address=77.90.31.18:4805 , piece_id=5CAGZW6RQYUUM76YBS5GWB76D7HPIN7B5M5NZBZLXJD2P5FL2A3Q, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.782757851 (540h49m49.653840886s) [noise=true , address=145.79.227.39:4013 , piece_id=3XU5U3GDHXTRAEHN5PUWLEKGGQAGO5ASDKG3QHAGBS2KJUPWDURQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:49.847377061 (540h49m49.718460096s) [noise=true , address=77.90.31.19:4841 , piece_id=KSQ7B5DSFPBS5RFWUKSCWRK6L7DCZJS5HUMIMZEGB6CGVHTVUKWQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:50.037733874 (540h49m49.908816909s) [noise=true , address=45.91.116.12:4225 , piece_id=B7PFPJM6JJ5K2EIR7ALTVWUHQF7TOSFOFDMGDO23LVRIJASTPPNQ, seg_pos={0 0}, upload=474300]: Done uploading piece. err: <nil> Dec 28 04:18:50.037752860 (540h49m49.908835905s) [seg_pos={0 0}, upload=474300]: Segment reached optimal threshold of 13 pieces. Dec 28 04:18:50.038011783 (540h49m49.909094828s) [noise=true , address=45.91.116.10:4084 , piece_id=P7AZAWCXJDXCKVJ7V7SGSZHF5VEAGMOEOTKNV5M62QOU4HL2ALRA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:59984->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:59984->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59984->45.91.116.10:4084: use of closed network connection Dec 28 04:18:50.038019138 (540h49m49.909102183s) [noise=true , address=45.91.116.10:4084 , piece_id=P7AZAWCXJDXCKVJ7V7SGSZHF5VEAGMOEOTKNV5M62QOU4HL2ALRA, seg_pos={0 0}, upload=474300]: Long tail cancellation due to optimal threshold Dec 28 04:18:50.038312466 (540h49m49.909395501s) [noise=true , address=145.79.227.33:4869 , piece_id=B2F6MM46WWJIXHVKZAJC5FQ27FQS73X2MUDFYVSTQJZNBHIN4EBA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 04:18:50.038319900 (540h49m49.909402945s) [noise=true , address=145.79.227.33:4869 , piece_id=B2F6MM46WWJIXHVKZAJC5FQ27FQS73X2MUDFYVSTQJZNBHIN4EBA, seg_pos={0 0}, upload=474300]: Long tail cancellation due to optimal threshold Dec 28 04:18:50.038450769 (540h49m49.909533804s) [noise=true , address=145.79.227.44:4195 , piece_id=4C6D5O454BO27S2MO5WZN656QTDVYOIY5CSMC7JGLU2YCUL3KCIA, seg_pos={0 0}, upload=474300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:18:50.038458353 (540h49m49.909541398s) [noise=true , address=145.79.227.44:4195 , piece_id=4C6D5O454BO27S2MO5WZN656QTDVYOIY5CSMC7JGLU2YCUL3KCIA, seg_pos={0 0}, upload=474300]: Long tail cancellation due to optimal threshold Dec 28 04:18:50.038479664 (540h49m49.909562699s) [seg_pos={0 0}, upload=474300]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:18:50.038565016 (540h49m49.909648051s) [upload=474300]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:18:50.038570337 (540h49m49.909653372s) [upload=474300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:18:50.328489033 (540h49m50.199572088s) [upload=474300]: Done uploading segments... Dec 28 04:18:50.328513149 (540h49m50.199596185s) [upload=474300]: Upload finished. err: <nil> Duration: 30.00300645s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)