Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.......................X...........XXX..................] duration: 6.070284461s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:59:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 6.070384832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.......................X...........XXX.................] duration: 5.769192228s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:58:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.769215283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.......................X...........XXX................] duration: 6.070169979s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:58:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 6.070211489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.......................X...........XXX...............] duration: 5.777329167s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:57:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.777353394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.......................X...........XXX..............] duration: 6.074117525s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:57:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 6.074147272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.......................X...........XXX.............] duration: 5.820608553s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:56:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.820722502s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.......................X...........XXX............] duration: 5.776970413s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:55:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.777001221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.......................X...........XXX...........] duration: 5.662595205s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:55:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.66264978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.......................X...........XXX..........] duration: 6.140431618s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:55:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 6.140456897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X.......................X...........XXX.........] duration: 5.654930889s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:54:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.654955416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.......................X...........XXX........] duration: 5.840515855s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:53:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.840555049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.......................X...........XXX.......] duration: 5.690798689s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:52:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.690828677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..............X.......................X...........XXX......] duration: 6.034815688s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:51:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 6.034840455s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..............X.......................X...........XXX.....] duration: 5.849736353s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:51:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.849767711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.X..............X.......................X...........XXX....] duration: 5.903625945s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:50:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.903647174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.X..............X.......................X...........XXX...] duration: 5.961459558s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:50:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.961481369s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.X..............X.......................X...........XXX..] duration: 5.802377239s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:49:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.802419018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.X..............X.......................X...........XXX.] duration: 5.777970418s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:47:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 5.777998462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.X..............X.......................X...........XXX] duration: 30.003831139s trace: https://jaeger.dev.storj.io/trace/5712756e3e35e904 FAIL (2025-12-25 10:47:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:47:18.017506319 (475h18m17.888589354s) [upload=417202]: Starting upload Dec 25 10:47:18.017564199 (475h18m17.888647234s) [upload=417202]: Done starting upload Dec 25 10:47:18.017659621 (475h18m17.888742656s) [upload=417202]: Uploading segments... Dec 25 10:47:18.017805910 (475h18m17.888888945s) [upload=417202]: Got next segment. Inline: false Dec 25 10:47:18.017840656 (475h18m17.888923701s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:47:18.017845355 (475h18m17.88892839s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:47:18.261677628 (475h18m18.132760663s) [seg_pos={0 0}, upload=417202]: Begin upload segment... Dec 25 10:47:18.261717374 (475h18m18.132800409s) [seg_pos={0 0}, upload=417202]: Done begin upload segment. Dec 25 10:47:18.261741199 (475h18m18.132824244s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261857310 (475h18m18.132940355s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.261969865 (475h18m18.1330529s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262397479 (475h18m18.133480524s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.262773385 (475h18m18.13385643s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263153549 (475h18m18.134236594s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.263528753 (475h18m18.134611788s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264028595 (475h18m18.13511163s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.264500012 (475h18m18.135583047s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.265325334 (475h18m18.136408369s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.266543995 (475h18m18.13762704s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.267592271 (475h18m18.138675306s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.268863812 (475h18m18.139946847s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.270958701 (475h18m18.142041736s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.275921501 (475h18m18.147004536s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:18.278258251 (475h18m18.149341286s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Uploading piece... Dec 25 10:47:19.977216509 (475h18m19.848299544s) [upload=417202]: Next returned nil segment Dec 25 10:47:19.977280271 (475h18m19.848363316s) [upload=417202]: Waiting for error group managing segments... Dec 25 10:47:20.272802982 (475h18m20.143886027s) [noise=true , address=145.79.227.38:4979 , piece_id=MAIJYQFWE76TI2HKUJYBKIWLLOJWEAAB2X4QK4QHEI6GGVUI66XQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.280809267 (475h18m20.151892302s) [noise=true , address=145.79.227.12:4765 , piece_id=IB5KR24KMJWCZGIB733MWXLYWNYKDG3AKLEA77CKYGOT22LFEKSQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.281124567 (475h18m20.152207603s) [noise=true , address=145.79.227.41:4085 , piece_id=MQKDROD4G2TMEA7UFF6BIFFQD6NYOJFQPTN2B3CZRLQG3JQY7CPA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.283881487 (475h18m20.154964532s) [noise=true , address=145.79.227.22:4509 , piece_id=5IRUOTFBYIXHYROEEBDH5XEOLMJDDLENV6BWC2GINTMUMAHT7E5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285049772 (475h18m20.156132807s) [noise=true , address=145.79.227.13:4802 , piece_id=AQAJHFEJEQT265XMDQGFKDDSG6IMNJXHHQNUOTXFZ3WPA4FOI7BA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285264931 (475h18m20.156347967s) [noise=true , address=145.79.227.50:4374 , piece_id=QGJIVVXIGTAQJS7IXAGTYJDFI4H2B25QPUJRFR3LNNAY5PXZH7KA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.285535016 (475h18m20.156618051s) [noise=true , address=145.79.227.19:4077 , piece_id=FKTUICQG2XYYHFP6U76AOA7N2WUJCIKSJXCMQNGWOBB23TJMPJJA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.294912983 (475h18m20.165996018s) [noise=true , address=145.79.227.45:4230 , piece_id=HSHI6KFODSJNT7DJIHFQCKACULI4A777MTKCTQTFRVRLCNGAL4IA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:20.505154696 (475h18m20.376237771s) [noise=true , address=45.91.116.12:4226 , piece_id=VL7WLEB5BMDMYL5G7DVVUBND4TKLCEZ3Y5YRURT4PP32Z2BU47LA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.053314697 (475h18m20.924397732s) [noise=true , address=45.91.116.13:4153 , piece_id=ONOFCGU4ICGGBBU5653MCHQN7AX54ICI275KJL6G6OCC5AHAWDAQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.198946442 (475h18m21.070029487s) [noise=true , address=45.91.116.8:4047 , piece_id=ELQJ3KOVMDN33LZWDKZVFXHA3FB4YTDBHSWQNOHDKFNX44STQJBA, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.277668521 (475h18m21.148751566s) [noise=true , address=45.91.116.11:4120 , piece_id=RYHMBA3PAPTMBHM5VSC4JKWMAEVPJJGFDLEFWJNT67NGBG4QK56A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352090943 (475h18m21.223173988s) [noise=true , address=45.91.116.10:4084 , piece_id=TNUZFGYOBDX5GPA3Q2WDETRVNJVLY37I2IGXVZTOJAGCWI7DWY5A, seg_pos={0 0}, upload=417202]: Done uploading piece. err: <nil> Dec 25 10:47:21.352117463 (475h18m21.223200508s) [seg_pos={0 0}, upload=417202]: Segment reached optimal threshold of 13 pieces. Dec 25 10:47:21.352373441 (475h18m21.223456496s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352385754 (475h18m21.223468799s) [noise=true , address=145.79.227.30:4761 , piece_id=SMQGO6I2YSUL5K2OHS2ARJ6SWNYURMK4COIUFQ32K2YH65JCIKQQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352514820 (475h18m21.223597875s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352535859 (475h18m21.223618894s) [noise=true , address=45.91.116.9:4263 , piece_id=ZXDCH4PPZ3RHTH7VUCLP6LD2FZVT5XUAQB3BJV4Z36MM2D4NLJTQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352705342 (475h18m21.223788377s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:47:21.352712897 (475h18m21.223795932s) [noise=true , address=77.90.31.10:4517 , piece_id=GSHVPVLSWNZDMIHR2ZRKXFUNQ6ZQEAHMOLWXXIUSUQNJMHHOXHRQ, seg_pos={0 0}, upload=417202]: Long tail cancellation due to optimal threshold Dec 25 10:47:21.352742202 (475h18m21.223825247s) [seg_pos={0 0}, upload=417202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:47:21.352819841 (475h18m21.223902876s) [upload=417202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:47:21.352824560 (475h18m21.223907605s) [upload=417202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:47:21.638799310 (475h18m21.509882355s) [upload=417202]: Done uploading segments... Dec 25 10:47:21.638820100 (475h18m21.509903145s) [upload=417202]: Upload finished. err: <nil> Duration: 30.003849465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.X..............X.......................X...........XX] duration: 30.002785845s trace: https://jaeger.dev.storj.io/trace/14c6733952a1f673 FAIL (2025-12-25 10:46:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: W3ZCLCVTO3RJUHWXOEGCH4PVRVQR6CDS6NWT3B3X7JODWA3HOU4Q) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: PRJY6FWGVA45XVIB2BYVC5TQWTHOBAR2ES4BSWRG66INGDYROCLA) context deadline exceeded ======LOGS====== Dec 25 10:46:03.645183905 (475h17m3.51626694s) [upload=417185]: Starting upload Dec 25 10:46:03.645242356 (475h17m3.516325401s) [upload=417185]: Done starting upload Dec 25 10:46:03.645350352 (475h17m3.516433397s) [upload=417185]: Uploading segments... Dec 25 10:46:03.645464529 (475h17m3.516547574s) [upload=417185]: Got next segment. Inline: false Dec 25 10:46:03.645513552 (475h17m3.516596607s) [upload=417185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:46:03.645519213 (475h17m3.516602258s) [upload=417185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:46:03.895050569 (475h17m3.766133634s) [seg_pos={0 0}, upload=417185]: Begin upload segment... Dec 25 10:46:03.895109140 (475h17m3.766192185s) [seg_pos={0 0}, upload=417185]: Done begin upload segment. Dec 25 10:46:03.895148155 (475h17m3.7662312s) [noise=true , address=145.79.227.6:4257 , piece_id=BMF3G3UUAEUFERN6MUYVTERRMAVDEHKT43LMC4R675VTYE64MDTA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.895292209 (475h17m3.766375254s) [noise=true , address=77.90.31.10:4517 , piece_id=54HHABZ22DYCZSVSNTPLBBVS6EZ27OCRKHBQJLQNDLXUHHSDAOSQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.895967885 (475h17m3.76705093s) [noise=true , address=145.79.227.19:4079 , piece_id=B5J7FJLJ2ZRFID35G57MIJTL6TVI7KFQAKPTQT2DWE4NB7JPPJEQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.896279839 (475h17m3.767362884s) [noise=true , address=77.90.31.13:4627 , piece_id=W3ZCLCVTO3RJUHWXOEGCH4PVRVQR6CDS6NWT3B3X7JODWA3HOU4Q, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.896572687 (475h17m3.767655732s) [noise=true , address=145.79.227.51:4410 , piece_id=WJHZKGZI5ZHGWLMTIPJ7IDDTEEHJEWS4RRQIYKAZJSAUS3LSHJWQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.896858922 (475h17m3.767941967s) [noise=true , address=145.79.227.30:4762 , piece_id=6QZUNES3O6VDPNZHYWFOPJOAGNVPUOJH6GBAZ7CJCRKPONUK3YDA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.897137353 (475h17m3.768220398s) [noise=true , address=145.79.227.29:4942 , piece_id=ZNQCE5XFLNBPLPLE4RM6JX2EBRFYMU4NME2CIZSIYVSTNJ5BJTQA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.897455238 (475h17m3.768538283s) [noise=true , address=145.79.227.12:4765 , piece_id=GCABNPP5CGX6S4WIM6NP4KMYXDNN7E6M75ADPGRIK37WVOVS2XZA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.899403458 (475h17m3.770486493s) [noise=true , address=145.79.227.7:4294 , piece_id=IFGGXVLBMTXDLVBJKQOZFAYOHUCYGTPUEE66ATS5D4FX3BLOXVFQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.900795017 (475h17m3.771878062s) [noise=true , address=145.79.227.48:4302 , piece_id=FP7NBQ2ILY7L7KRQ26NLMBSHTCFLPL2NFU5L5IZXJ3XTL6LHBFZQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.902252162 (475h17m3.773335197s) [noise=true , address=77.90.31.11:4554 , piece_id=PRJY6FWGVA45XVIB2BYVC5TQWTHOBAR2ES4BSWRG66INGDYROCLA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.904673953 (475h17m3.775756988s) [noise=true , address=145.79.227.25:4618 , piece_id=DFMBHHPDKZHPFYF73SWN5BB22SYOFQUMJFUWUK65PQL2TUZGALMQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.906246407 (475h17m3.777329452s) [noise=true , address=45.91.116.14:4192 , piece_id=BU745DZPWUPHZSQV67KBBVTCIYX3NOSUEUZCONHKLRYF7BUVWEGA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.907591038 (475h17m3.778674073s) [noise=true , address=145.79.227.46:4601 , piece_id=QWPOBUQGQZHBLJTPQ5PFO2TX7BE45BFZ3AWNCHSGCRZAVCZSMMQA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.908854654 (475h17m3.779937699s) [noise=true , address=145.79.227.43:4158 , piece_id=RGMWBHRCVKOAOWO26UOY4HBPN2UAHA3JTACIG6ZP7LKAYWMG4DVA, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:03.910108662 (475h17m3.781191707s) [noise=true , address=145.79.227.47:4265 , piece_id=ZSVY4N2QBE6NHB336L6UGYJKQ4HBHBTOHUHM7WW5WGOP4DSJ5SSQ, seg_pos={0 0}, upload=417185]: Uploading piece... Dec 25 10:46:05.619526557 (475h17m5.490609622s) [upload=417185]: Next returned nil segment Dec 25 10:46:05.619555642 (475h17m5.490638677s) [upload=417185]: Waiting for error group managing segments... Dec 25 10:46:05.909330237 (475h17m5.780413283s) [noise=true , address=145.79.227.25:4618 , piece_id=DFMBHHPDKZHPFYF73SWN5BB22SYOFQUMJFUWUK65PQL2TUZGALMQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.916522563 (475h17m5.787605608s) [noise=true , address=145.79.227.46:4601 , piece_id=QWPOBUQGQZHBLJTPQ5PFO2TX7BE45BFZ3AWNCHSGCRZAVCZSMMQA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.922864139 (475h17m5.793947184s) [noise=true , address=145.79.227.51:4410 , piece_id=WJHZKGZI5ZHGWLMTIPJ7IDDTEEHJEWS4RRQIYKAZJSAUS3LSHJWQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.925332168 (475h17m5.796415203s) [noise=true , address=145.79.227.48:4302 , piece_id=FP7NBQ2ILY7L7KRQ26NLMBSHTCFLPL2NFU5L5IZXJ3XTL6LHBFZQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.929558896 (475h17m5.800641941s) [noise=true , address=145.79.227.19:4079 , piece_id=B5J7FJLJ2ZRFID35G57MIJTL6TVI7KFQAKPTQT2DWE4NB7JPPJEQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.934438628 (475h17m5.805521663s) [noise=true , address=145.79.227.12:4765 , piece_id=GCABNPP5CGX6S4WIM6NP4KMYXDNN7E6M75ADPGRIK37WVOVS2XZA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.936452463 (475h17m5.807535498s) [noise=true , address=145.79.227.7:4294 , piece_id=IFGGXVLBMTXDLVBJKQOZFAYOHUCYGTPUEE66ATS5D4FX3BLOXVFQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.938350758 (475h17m5.809433803s) [noise=true , address=145.79.227.43:4158 , piece_id=RGMWBHRCVKOAOWO26UOY4HBPN2UAHA3JTACIG6ZP7LKAYWMG4DVA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.941483022 (475h17m5.812566077s) [noise=true , address=145.79.227.29:4942 , piece_id=ZNQCE5XFLNBPLPLE4RM6JX2EBRFYMU4NME2CIZSIYVSTNJ5BJTQA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.962315901 (475h17m5.833398936s) [noise=true , address=145.79.227.47:4265 , piece_id=ZSVY4N2QBE6NHB336L6UGYJKQ4HBHBTOHUHM7WW5WGOP4DSJ5SSQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:05.986971289 (475h17m5.858054334s) [noise=true , address=145.79.227.6:4257 , piece_id=BMF3G3UUAEUFERN6MUYVTERRMAVDEHKT43LMC4R675VTYE64MDTA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:06.100560449 (475h17m5.971643494s) [noise=true , address=77.90.31.13:4627 , piece_id=W3ZCLCVTO3RJUHWXOEGCH4PVRVQR6CDS6NWT3B3X7JODWA3HOU4Q, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:06.173289307 (475h17m6.044372343s) [noise=true , address=77.90.31.11:4554 , piece_id=PRJY6FWGVA45XVIB2BYVC5TQWTHOBAR2ES4BSWRG66INGDYROCLA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: <nil> Dec 25 10:46:06.173312711 (475h17m6.044395746s) [seg_pos={0 0}, upload=417185]: Segment reached optimal threshold of 13 pieces. Dec 25 10:46:06.173523022 (475h17m6.044606067s) [noise=true , address=145.79.227.30:4762 , piece_id=6QZUNES3O6VDPNZHYWFOPJOAGNVPUOJH6GBAZ7CJCRKPONUK3YDA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 25 10:46:06.173530988 (475h17m6.044614033s) [noise=true , address=145.79.227.30:4762 , piece_id=6QZUNES3O6VDPNZHYWFOPJOAGNVPUOJH6GBAZ7CJCRKPONUK3YDA, seg_pos={0 0}, upload=417185]: Long tail cancellation due to optimal threshold Dec 25 10:46:06.173618043 (475h17m6.044701078s) [noise=true , address=45.91.116.14:4192 , piece_id=BU745DZPWUPHZSQV67KBBVTCIYX3NOSUEUZCONHKLRYF7BUVWEGA, seg_pos={0 0}, upload=417185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:58040->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:58040->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58040->45.91.116.14:4192: use of closed network connection Dec 25 10:46:06.173629775 (475h17m6.04471281s) [noise=true , address=45.91.116.14:4192 , piece_id=BU745DZPWUPHZSQV67KBBVTCIYX3NOSUEUZCONHKLRYF7BUVWEGA, seg_pos={0 0}, upload=417185]: Long tail cancellation due to optimal threshold Dec 25 10:46:06.174036830 (475h17m6.045119865s) [noise=true , address=77.90.31.10:4517 , piece_id=54HHABZ22DYCZSVSNTPLBBVS6EZ27OCRKHBQJLQNDLXUHHSDAOSQ, seg_pos={0 0}, upload=417185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 10:46:06.174044815 (475h17m6.04512786s) [noise=true , address=77.90.31.10:4517 , piece_id=54HHABZ22DYCZSVSNTPLBBVS6EZ27OCRKHBQJLQNDLXUHHSDAOSQ, seg_pos={0 0}, upload=417185]: Long tail cancellation due to optimal threshold Dec 25 10:46:06.174066768 (475h17m6.045149813s) [seg_pos={0 0}, upload=417185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:46:06.174192768 (475h17m6.045275833s) [upload=417185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:46:06.174200823 (475h17m6.045283858s) [upload=417185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:46:06.466033598 (475h17m6.337116633s) [upload=417185]: Done uploading segments... Dec 25 10:46:06.466053907 (475h17m6.337136952s) [upload=417185]: Upload finished. err: <nil> Duration: 30.002809791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.X..............X.......................X...........X] duration: 30.003685319s trace: https://jaeger.dev.storj.io/trace/5159ecaf1e8f769b FAIL (2025-12-25 10:45:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:45:21.720192346 (475h16m21.591275391s) [upload=417175]: Starting upload Dec 25 10:45:21.720254234 (475h16m21.591337269s) [upload=417175]: Done starting upload Dec 25 10:45:21.720534858 (475h16m21.591617893s) [upload=417175]: Uploading segments... Dec 25 10:45:21.720580615 (475h16m21.59166365s) [upload=417175]: Got next segment. Inline: false Dec 25 10:45:21.720667140 (475h16m21.591750185s) [upload=417175]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:45:21.720672040 (475h16m21.591755085s) [upload=417175]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:45:21.969061180 (475h16m21.840144225s) [seg_pos={0 0}, upload=417175]: Begin upload segment... Dec 25 10:45:21.969163715 (475h16m21.84024675s) [seg_pos={0 0}, upload=417175]: Done begin upload segment. Dec 25 10:45:21.969219021 (475h16m21.840302066s) [noise=true , address=45.91.116.11:4119 , piece_id=R4QSAYXV72U5OXU3J6U5C5FUC4MDRZAA6DYZDKD3OGUNXRAJCKPA, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.969242807 (475h16m21.840325842s) [noise=true , address=145.79.227.52:4446 , piece_id=MWVPORRBN4YEKYGCPZIDH7WLX7QIWX63WJRPREKPZUTWZC6AELHQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.970182074 (475h16m21.841265119s) [noise=true , address=45.91.116.12:4228 , piece_id=EDNHWBC2DW2XB56N4ALRYTKZCGWXHTHGEDMGZR5F2ZV645FPKZOA, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.970279560 (475h16m21.841362595s) [noise=true , address=145.79.227.48:4303 , piece_id=VJHEP3JVATB34BI2GDE2XE6NSRDK4R2P47C22W7NJCRIB4CGW66A, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.970653412 (475h16m21.841736457s) [noise=true , address=45.91.116.7:4010 , piece_id=R2G5JLUDV4RTB4UBJBCCRLYYBARHB3M6JWS7RFMI3GXC4FSHE6GA, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.970672479 (475h16m21.841755524s) [noise=true , address=45.91.116.14:4191 , piece_id=CKZHSYM36LQUNKJIVSD4QUZHZLDFNVWN24TWUTPHMORUP2PAFGQQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.970994862 (475h16m21.842077908s) [noise=true , address=145.79.227.32:4835 , piece_id=DMC6NUO2W65OBYACK5HQSNIV4CF5TBHS2S7DZG6BEYVFRVKCERFA, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.971040249 (475h16m21.842123294s) [noise=true , address=145.79.227.16:4910 , piece_id=YFGIOIB4LN2TSLUKGII7ADJQKY6NXCPQVLH7OA3BFZZQBSONECMQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.971327696 (475h16m21.842410731s) [noise=true , address=145.79.227.33:4869 , piece_id=SAMY3FPLPJXND6BKGLDH5D74MUKPHVBEM56PF3GJYCB7UGWPRA6Q, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.971511246 (475h16m21.842594281s) [noise=true , address=145.79.227.22:4509 , piece_id=5XI7ETTPCV5S7KOSNLO7M4WZAGSTXRVVNURF3NWIBRMH2RA4AKWQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.971657825 (475h16m21.84274086s) [noise=true , address=45.91.116.5:4297 , piece_id=QTACZCRNIMRRCQ77YDF2XKGZ36KGXMKMYWHJHNT2KZEPBOLNKLTA, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.972037909 (475h16m21.843120954s) [noise=true , address=145.79.227.3:4150 , piece_id=4EL6LO7D2POLENHTQRAQSTBFDT7ZWISYIP5QH6EYKZOLQWWD6YPQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.972433782 (475h16m21.843516817s) [noise=true , address=145.79.227.29:4941 , piece_id=5APUJOOGHBTN2KAA6ZNMDTNSNAP7YNA3PNLBAXKWVVK7TG2RDGAQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.972829236 (475h16m21.843912271s) [noise=true , address=45.91.116.4:4940 , piece_id=24NQTUS5EXHS2UOWKFVHTX6FFS2DPBQCIEM4KR6NARGZXGH2ABGQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.973180825 (475h16m21.84426387s) [noise=true , address=145.79.227.39:4015 , piece_id=PQBSNMUTFEFCWX7VDYB3GAJZP67QOWTQFMFJH6NTFTFAGA62LE3Q, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:21.973371698 (475h16m21.844454733s) [noise=true , address=145.79.227.13:4804 , piece_id=Q2OHOUWELLXVDJLOV2PK73VRYRM2I4RKRBR3M3XBNK52LDOG2VQQ, seg_pos={0 0}, upload=417175]: Uploading piece... Dec 25 10:45:23.595258401 (475h16m23.466341436s) [upload=417175]: Next returned nil segment Dec 25 10:45:23.595322834 (475h16m23.466405899s) [upload=417175]: Waiting for error group managing segments... Dec 25 10:45:23.979830797 (475h16m23.850913842s) [noise=true , address=145.79.227.39:4015 , piece_id=PQBSNMUTFEFCWX7VDYB3GAJZP67QOWTQFMFJH6NTFTFAGA62LE3Q, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.980192605 (475h16m23.85127565s) [noise=true , address=145.79.227.13:4804 , piece_id=Q2OHOUWELLXVDJLOV2PK73VRYRM2I4RKRBR3M3XBNK52LDOG2VQQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.981783164 (475h16m23.852866199s) [noise=true , address=145.79.227.29:4941 , piece_id=5APUJOOGHBTN2KAA6ZNMDTNSNAP7YNA3PNLBAXKWVVK7TG2RDGAQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.985386996 (475h16m23.856470041s) [noise=true , address=145.79.227.32:4835 , piece_id=DMC6NUO2W65OBYACK5HQSNIV4CF5TBHS2S7DZG6BEYVFRVKCERFA, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.986596369 (475h16m23.857679414s) [noise=true , address=145.79.227.16:4910 , piece_id=YFGIOIB4LN2TSLUKGII7ADJQKY6NXCPQVLH7OA3BFZZQBSONECMQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.988116814 (475h16m23.859199849s) [noise=true , address=145.79.227.3:4150 , piece_id=4EL6LO7D2POLENHTQRAQSTBFDT7ZWISYIP5QH6EYKZOLQWWD6YPQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.988571389 (475h16m23.859654434s) [noise=true , address=145.79.227.22:4509 , piece_id=5XI7ETTPCV5S7KOSNLO7M4WZAGSTXRVVNURF3NWIBRMH2RA4AKWQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:23.990639247 (475h16m23.861722282s) [noise=true , address=145.79.227.33:4869 , piece_id=SAMY3FPLPJXND6BKGLDH5D74MUKPHVBEM56PF3GJYCB7UGWPRA6Q, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:24.001703456 (475h16m23.872786501s) [noise=true , address=145.79.227.48:4303 , piece_id=VJHEP3JVATB34BI2GDE2XE6NSRDK4R2P47C22W7NJCRIB4CGW66A, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:24.002585755 (475h16m23.87366879s) [noise=true , address=145.79.227.52:4446 , piece_id=MWVPORRBN4YEKYGCPZIDH7WLX7QIWX63WJRPREKPZUTWZC6AELHQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:24.200065527 (475h16m24.071148562s) [noise=true , address=45.91.116.4:4940 , piece_id=24NQTUS5EXHS2UOWKFVHTX6FFS2DPBQCIEM4KR6NARGZXGH2ABGQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:24.327353743 (475h16m24.198436778s) [noise=true , address=45.91.116.11:4119 , piece_id=R4QSAYXV72U5OXU3J6U5C5FUC4MDRZAA6DYZDKD3OGUNXRAJCKPA, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:24.347264416 (475h16m24.218347452s) [noise=true , address=45.91.116.14:4191 , piece_id=CKZHSYM36LQUNKJIVSD4QUZHZLDFNVWN24TWUTPHMORUP2PAFGQQ, seg_pos={0 0}, upload=417175]: Done uploading piece. err: <nil> Dec 25 10:45:24.347283723 (475h16m24.218366758s) [seg_pos={0 0}, upload=417175]: Segment reached optimal threshold of 13 pieces. Dec 25 10:45:24.347464588 (475h16m24.218547623s) [noise=true , address=45.91.116.5:4297 , piece_id=QTACZCRNIMRRCQ77YDF2XKGZ36KGXMKMYWHJHNT2KZEPBOLNKLTA, seg_pos={0 0}, upload=417175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 25 10:45:24.347471040 (475h16m24.218554085s) [noise=true , address=45.91.116.5:4297 , piece_id=QTACZCRNIMRRCQ77YDF2XKGZ36KGXMKMYWHJHNT2KZEPBOLNKLTA, seg_pos={0 0}, upload=417175]: Long tail cancellation due to optimal threshold Dec 25 10:45:24.347684817 (475h16m24.218767862s) [noise=true , address=45.91.116.12:4228 , piece_id=EDNHWBC2DW2XB56N4ALRYTKZCGWXHTHGEDMGZR5F2ZV645FPKZOA, seg_pos={0 0}, upload=417175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:45:24.347693233 (475h16m24.218776268s) [noise=true , address=45.91.116.12:4228 , piece_id=EDNHWBC2DW2XB56N4ALRYTKZCGWXHTHGEDMGZR5F2ZV645FPKZOA, seg_pos={0 0}, upload=417175]: Long tail cancellation due to optimal threshold Dec 25 10:45:24.347752887 (475h16m24.218835922s) [noise=true , address=45.91.116.7:4010 , piece_id=R2G5JLUDV4RTB4UBJBCCRLYYBARHB3M6JWS7RFMI3GXC4FSHE6GA, seg_pos={0 0}, upload=417175]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 25 10:45:24.347767163 (475h16m24.218850198s) [noise=true , address=45.91.116.7:4010 , piece_id=R2G5JLUDV4RTB4UBJBCCRLYYBARHB3M6JWS7RFMI3GXC4FSHE6GA, seg_pos={0 0}, upload=417175]: Long tail cancellation due to optimal threshold Dec 25 10:45:24.347804374 (475h16m24.218887409s) [seg_pos={0 0}, upload=417175]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:45:24.347996090 (475h16m24.219079125s) [upload=417175]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:45:24.348002842 (475h16m24.219085877s) [upload=417175]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:45:24.634659120 (475h16m24.505742165s) [upload=417175]: Done uploading segments... Dec 25 10:45:24.634682264 (475h16m24.505765309s) [upload=417175]: Upload finished. err: <nil> Duration: 30.00372248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.X..............X.......................X...........] duration: 5.777986207s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:44:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.778012707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.X..............X.......................X..........] duration: 5.980897753s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:43:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.980915987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.X..............X.......................X.........] duration: 5.664987269s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:43:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.665017687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.X..............X.......................X........] duration: 6.051948923s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:43:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 6.051979962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.X..............X.......................X.......] duration: 6.045380036s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:41:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 6.045409994s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.X..............X.......................X......] duration: 5.767112966s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:41:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.7671407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.X..............X.......................X.....] duration: 5.705655559s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:40:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.705683782s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.X..............X.......................X....] duration: 5.814839174s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:39:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.81486435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.X..............X.......................X...] duration: 5.900979007s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:38:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.901007301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.X..............X.......................X..] duration: 5.814040349s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:37:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 5.814064856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.X..............X.......................X.] duration: 27.955808766s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:37:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 27.955926611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.X..............X.......................X] duration: 30.002749438s trace: https://jaeger.dev.storj.io/trace/2eb6d03a313e329a FAIL (2025-12-25 10:36:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 25 10:36:30.374149002 (475h7m30.245232037s) [upload=417048]: Starting upload Dec 25 10:36:30.374259363 (475h7m30.245342398s) [upload=417048]: Done starting upload Dec 25 10:36:30.374408346 (475h7m30.245491391s) [upload=417048]: Uploading segments... Dec 25 10:36:30.374495381 (475h7m30.245578417s) [upload=417048]: Got next segment. Inline: false Dec 25 10:36:30.374557861 (475h7m30.245640896s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:36:30.374566367 (475h7m30.245649402s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:36:30.950175924 (475h7m30.821258969s) [seg_pos={0 0}, upload=417048]: Begin upload segment... Dec 25 10:36:30.950234777 (475h7m30.821317822s) [seg_pos={0 0}, upload=417048]: Done begin upload segment. Dec 25 10:36:30.950272057 (475h7m30.821355093s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950277989 (475h7m30.821361024s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.950383129 (475h7m30.821466164s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951116675 (475h7m30.822199721s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951477333 (475h7m30.822560378s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951483555 (475h7m30.8225666s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.951976513 (475h7m30.823059558s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952575654 (475h7m30.823658699s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.952814248 (475h7m30.823897293s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953092298 (475h7m30.824175343s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.953616796 (475h7m30.824699831s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954201460 (475h7m30.825284495s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954575392 (475h7m30.825658437s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.954933474 (475h7m30.826016519s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955317084 (475h7m30.826400119s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:30.955790035 (475h7m30.82687307s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Uploading piece... Dec 25 10:36:32.858915734 (475h7m32.729998769s) [upload=417048]: Next returned nil segment Dec 25 10:36:32.858928518 (475h7m32.730011563s) [upload=417048]: Waiting for error group managing segments... Dec 25 10:36:33.170071642 (475h7m33.041154677s) [noise=true , address=145.79.227.17:4971 , piece_id=EZGGCXK5S5TWN5T32DCH7G2JMRI4FKKSH7E3WP3SSH4QKOFOQMPA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.199852369 (475h7m33.070935404s) [noise=true , address=145.79.227.50:4374 , piece_id=GIR65SW3HUI6H5XZQRHQDGI6DSF2AZN2TD64IBKJI74ECAOQ62GA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:33.547164414 (475h7m33.418247449s) [noise=true , address=45.91.116.12:4226 , piece_id=XRLQQDFDX4MMOQ4F3DQ5DWQA374BP7GVFCQN5BXH4X7NFJNWSBCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.300335285 (475h7m34.17141833s) [noise=true , address=145.79.227.47:4266 , piece_id=M6FOCYYHH7CRG62AHFH4Y7FF2TTLGOHMPTG37333YVEUI6TA3JZA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.666069658 (475h7m34.537152713s) [noise=true , address=145.79.227.46:4603 , piece_id=GSPNMWGAFKH5BARLZ57XIVQBYRXVNOLAPMOSGG4CAYIFUS6SYUWA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.932732825 (475h7m34.80381586s) [noise=true , address=145.79.227.5:4221 , piece_id=E72KKOGT7WAXYI2W5NWMGNX7U4UYK6CBLFQ4M2QQDQFT2XVTPGCA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:34.942206264 (475h7m34.813289309s) [noise=true , address=145.79.227.25:4618 , piece_id=54FV7JSKNKMDKE7363RZ52GWUZJGSEQ37DVNWPOATX6ACMHXTKQA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.033910242 (475h7m34.904993277s) [noise=true , address=45.91.116.6:4973 , piece_id=4PBBXUVTAH7WGCN2VHT6LWCXPHNIL7SME6DSXVKYQWZYDDABEHHA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.293062046 (475h7m35.164145091s) [noise=true , address=145.79.227.30:4762 , piece_id=RIVHMW4FVHKOJ5MGVSJHW4YH4H2YAMACM6M2CSZ4BJELACO66GMQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:35.380095441 (475h7m35.251178476s) [noise=true , address=145.79.227.34:4907 , piece_id=ROR2WORQDZZ46V4FDHO3ZWQGPCY4NRMSZAS63SCHAGOXOOWFLJDQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.165219937 (475h7m36.036302972s) [noise=true , address=145.79.227.8:4330 , piece_id=C5BZ3SMGEDLP65BZTJQQMA3RYYYNPNL2VURE42ZKDHDW7KDIMPAQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:36.765949627 (475h7m36.637032662s) [noise=true , address=145.79.227.35:4481 , piece_id=QWC7J3QLG2X4NPVU4YP4LS4WBG5JBM3HQP7Q7KPXBZ3YH4HNYEXA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385615294 (475h7m37.256698329s) [noise=true , address=145.79.227.40:4051 , piece_id=2ZTLWN2DMYVEIOHP5VNJUA5K5RUFOZ53B4S2UC2ZFOYBVTWIW2MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: <nil> Dec 25 10:36:37.385636154 (475h7m37.256719199s) [seg_pos={0 0}, upload=417048]: Segment reached optimal threshold of 13 pieces. Dec 25 10:36:37.386163227 (475h7m37.257246272s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:42840->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:36:37.386172675 (475h7m37.25725572s) [noise=true , address=145.79.227.16:4910 , piece_id=A42R4DXDNEGJLRN4HGVNWZ6P5ESBIOKHQEK7BXBG2GY6QSGYMKEQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386190700 (475h7m37.257273735s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 25 10:36:37.386230385 (475h7m37.25731342s) [noise=true , address=145.79.227.51:4410 , piece_id=ATW57YP4H4FHUCNMQAXKINAWPOAJGZU3LRROLOLH4IMTYEBLMFLA, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386589449 (475h7m37.257672484s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: read tcp 188.166.225.170:58674->145.79.227.13:4804: use of closed network connection; context canceled Dec 25 10:36:37.386599378 (475h7m37.257682423s) [noise=true , address=145.79.227.13:4804 , piece_id=VLYU673BS4AEPAMHVLGJPIEWPTOUGGERIV3ANNI4DXS6FWKPB3MQ, seg_pos={0 0}, upload=417048]: Long tail cancellation due to optimal threshold Dec 25 10:36:37.386642019 (475h7m37.257725054s) [seg_pos={0 0}, upload=417048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:36:37.386789390 (475h7m37.257872435s) [upload=417048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:36:37.386798617 (475h7m37.257881662s) [upload=417048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:36:37.703760150 (475h7m37.574843185s) [upload=417048]: Done uploading segments... Dec 25 10:36:37.703779556 (475h7m37.574862592s) [upload=417048]: Upload finished. err: <nil> Duration: 30.002767552s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.X..............X.......................] duration: 5.694226556s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:35:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.694303563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.X..............X......................] duration: 5.994075634s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:35:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.994107544s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.X..............X.....................] duration: 5.688924322s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:34:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.688953376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.X..............X....................] duration: 5.894157018s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:33:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.894182828s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.X..............X...................] duration: 5.945309147s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:32:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.945334135s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.X..............X..................] duration: 5.686474746s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:32:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.686505453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.X..............X.................] duration: 6.017437951s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:31:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 6.017461356s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.X..............X................] duration: 5.773079751s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:31:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.773110512s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..........................X.X..............X...............] duration: 5.788384631s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:30:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.788426902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX..........................X.X..............X..............] duration: 5.988765334s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:29:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.988790312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX..........................X.X..............X.............] duration: 5.715595373s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:29:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.715618989s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX..........................X.X..............X............] duration: 6.101551745s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:28:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 6.101588766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX..........................X.X..............X...........] duration: 5.877297444s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:27:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.877320406s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX..........................X.X..............X..........] duration: 5.710913754s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:26:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.710936987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XX..........................X.X..............X.........] duration: 6.043976064s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:25:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 6.043996523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XX..........................X.X..............X........] duration: 5.737975714s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:24:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.73800008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......XX..........................X.X..............X.......] duration: 6.159298916s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:23:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 6.159330907s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)