Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X.......XXX......X............X.........X..........] duration: 44.396399086s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 10:54:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 44.396415777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X.X.XX....XX.......X.X.....X..X....X..X..X.......XX.XX..X] duration: 30.002898564s trace: https://jaeger.dev.storj.io/trace/277543ab17dc0277 FAIL (2025-12-26 10:54:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: WFVAK5WOAKB7MU6OXGAEKIJZJ7752H6CMDM7FUOJ346C6QORWCEA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: KAEKPENETRQKTWPF7TPUXCHGFVQEBKWNIZIIGQ2KZDJY3WUKFR6Q) context deadline exceeded ======LOGS====== Dec 26 10:53:30.876061316 (499h24m30.747144351s) [upload=438316]: Starting upload Dec 26 10:53:30.876116632 (499h24m30.747199667s) [upload=438316]: Done starting upload Dec 26 10:53:30.876293398 (499h24m30.747376443s) [upload=438316]: Uploading segments... Dec 26 10:53:30.876335248 (499h24m30.747418283s) [upload=438316]: Got next segment. Inline: false Dec 26 10:53:30.876372398 (499h24m30.747455443s) [upload=438316]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:53:30.876377267 (499h24m30.747460302s) [upload=438316]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:53:31.118902305 (499h24m30.98998535s) [seg_pos={0 0}, upload=438316]: Begin upload segment... Dec 26 10:53:31.118955496 (499h24m30.990038531s) [seg_pos={0 0}, upload=438316]: Done begin upload segment. Dec 26 10:53:31.119015180 (499h24m30.990098225s) [noise=true , address=145.79.227.47:4265 , piece_id=YNRF4BAISN7RZJ73IX7AEEEMMVN66ZDB4ARCNOXG4CFJNCWL7EDA, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.119030499 (499h24m30.990113544s) [noise=true , address=45.91.116.5:4300 , piece_id=ZIEPSK6ZNUGYBHDWYIAW3ZRHFCWY3UZ5BCNQEHX7AFYNJITZBLPA, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.119640631 (499h24m30.990723666s) [noise=true , address=145.79.227.13:4803 , piece_id=YAJMHH5IZFWYKUCIPHA57UUASPYBOG36JNIQ7VVEM56HFKXXVF5Q, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.119939831 (499h24m30.991022866s) [noise=true , address=145.79.227.11:4732 , piece_id=65RK577VTXBZAJ27QNENACO7C5Q7VGX7VEABOGV72JJU57Z273GQ, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.120111698 (499h24m30.991194743s) [noise=true , address=145.79.227.35:4482 , piece_id=GLMP5DC24SRG7WGCQMORJZJ4NNFSVTF4A7CVCINQTL2ROE3PJVZA, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.120350232 (499h24m30.991433268s) [noise=true , address=45.91.116.13:4153 , piece_id=KAEKPENETRQKTWPF7TPUXCHGFVQEBKWNIZIIGQ2KZDJY3WUKFR6Q, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.120580070 (499h24m30.991663115s) [noise=true , address=145.79.227.15:4874 , piece_id=A7N5PITRMCNZBQZ5ELIUOLILWA5PRQCLRQGL7VFWXGLOX7J4OIQQ, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.120725206 (499h24m30.991808251s) [noise=true , address=145.79.227.39:4014 , piece_id=QOHXF6IB6CZF36OIQA5KFX3P4QWY2CBQAMTOPSVWAJ7JF5QXYUFQ, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.121108786 (499h24m30.992191831s) [noise=true , address=77.90.31.15:4697 , piece_id=WRIAEZ7TI5LCOEGTQOGT6OSEQFIFQ4UFYKKKTGPEOARECT2CCDSA, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.121283128 (499h24m30.992366173s) [noise=true , address=45.91.116.9:4263 , piece_id=RQFXEENVR2OGDX4O3LTJSXCQ2UZWD7PVFTMX3THJFE56VB5JALVA, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.121713198 (499h24m30.992796243s) [noise=true , address=145.79.227.40:4050 , piece_id=77KL7SNUQFSLS5HEWVOBMF6RIYSCEYKWTWWAV55O5YGNHBRXRUEA, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.121785746 (499h24m30.992868791s) [noise=true , address=145.79.227.41:4087 , piece_id=MEHHMABQE3XMNQWQJRRH54I4IFZES4TOFRBJ3NPQUHRDCKH3WUYQ, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.122072051 (499h24m30.993155096s) [noise=true , address=145.79.227.49:4337 , piece_id=Y5NLLLMCDHP57U427TCQZO3QK45KMAHKIMVXGE3SXCMOSNELHDZA, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.122492431 (499h24m30.993575466s) [noise=true , address=45.91.116.6:4974 , piece_id=WFVAK5WOAKB7MU6OXGAEKIJZJ7752H6CMDM7FUOJ346C6QORWCEA, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.122605226 (499h24m30.993688271s) [noise=true , address=45.91.116.8:4046 , piece_id=QIQCDDSKN2B7PENWNKW6AXH4DUWRXQQQ3J35GHB43XH5R7BHADKQ, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:31.123175913 (499h24m30.994258948s) [noise=true , address=145.79.227.37:4567 , piece_id=7CU5V3YQQ3YPIIUDDRVSAD22WR3NUMYGFGZ7CK6YXYIQBRDG6VJQ, seg_pos={0 0}, upload=438316]: Uploading piece... Dec 26 10:53:32.515706120 (499h24m32.386789165s) [noise=true , address=77.90.31.15:4697 , piece_id=WRIAEZ7TI5LCOEGTQOGT6OSEQFIFQ4UFYKKKTGPEOARECT2CCDSA, seg_pos={0 0}, upload=438316]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 26 10:53:32.838126090 (499h24m32.709209135s) [upload=438316]: Next returned nil segment Dec 26 10:53:32.838143183 (499h24m32.709226218s) [upload=438316]: Waiting for error group managing segments... Dec 26 10:53:33.125956237 (499h24m32.997039272s) [noise=true , address=145.79.227.39:4014 , piece_id=QOHXF6IB6CZF36OIQA5KFX3P4QWY2CBQAMTOPSVWAJ7JF5QXYUFQ, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.129081468 (499h24m33.000164513s) [noise=true , address=145.79.227.13:4803 , piece_id=YAJMHH5IZFWYKUCIPHA57UUASPYBOG36JNIQ7VVEM56HFKXXVF5Q, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.133878782 (499h24m33.004961817s) [noise=true , address=145.79.227.49:4337 , piece_id=Y5NLLLMCDHP57U427TCQZO3QK45KMAHKIMVXGE3SXCMOSNELHDZA, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.135295841 (499h24m33.006378876s) [noise=true , address=145.79.227.35:4482 , piece_id=GLMP5DC24SRG7WGCQMORJZJ4NNFSVTF4A7CVCINQTL2ROE3PJVZA, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.138064673 (499h24m33.009147718s) [noise=true , address=145.79.227.47:4265 , piece_id=YNRF4BAISN7RZJ73IX7AEEEMMVN66ZDB4ARCNOXG4CFJNCWL7EDA, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.142693146 (499h24m33.013776181s) [noise=true , address=145.79.227.40:4050 , piece_id=77KL7SNUQFSLS5HEWVOBMF6RIYSCEYKWTWWAV55O5YGNHBRXRUEA, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.153870690 (499h24m33.024953725s) [noise=true , address=145.79.227.37:4567 , piece_id=7CU5V3YQQ3YPIIUDDRVSAD22WR3NUMYGFGZ7CK6YXYIQBRDG6VJQ, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.156776584 (499h24m33.027859619s) [noise=true , address=145.79.227.41:4087 , piece_id=MEHHMABQE3XMNQWQJRRH54I4IFZES4TOFRBJ3NPQUHRDCKH3WUYQ, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.158747326 (499h24m33.029830361s) [noise=true , address=145.79.227.15:4874 , piece_id=A7N5PITRMCNZBQZ5ELIUOLILWA5PRQCLRQGL7VFWXGLOX7J4OIQQ, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.169223985 (499h24m33.04030703s) [noise=true , address=145.79.227.11:4732 , piece_id=65RK577VTXBZAJ27QNENACO7C5Q7VGX7VEABOGV72JJU57Z273GQ, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.403639431 (499h24m33.274722476s) [noise=true , address=45.91.116.13:4153 , piece_id=KAEKPENETRQKTWPF7TPUXCHGFVQEBKWNIZIIGQ2KZDJY3WUKFR6Q, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.424986200 (499h24m33.296069235s) [noise=true , address=45.91.116.5:4300 , piece_id=ZIEPSK6ZNUGYBHDWYIAW3ZRHFCWY3UZ5BCNQEHX7AFYNJITZBLPA, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.458217086 (499h24m33.329300121s) [noise=true , address=45.91.116.6:4974 , piece_id=WFVAK5WOAKB7MU6OXGAEKIJZJ7752H6CMDM7FUOJ346C6QORWCEA, seg_pos={0 0}, upload=438316]: Done uploading piece. err: <nil> Dec 26 10:53:33.458241934 (499h24m33.329324979s) [seg_pos={0 0}, upload=438316]: Segment reached optimal threshold of 13 pieces. Dec 26 10:53:33.458586450 (499h24m33.329669495s) [noise=true , address=45.91.116.8:4046 , piece_id=QIQCDDSKN2B7PENWNKW6AXH4DUWRXQQQ3J35GHB43XH5R7BHADKQ, seg_pos={0 0}, upload=438316]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 26 10:53:33.458595677 (499h24m33.329678722s) [noise=true , address=45.91.116.8:4046 , piece_id=QIQCDDSKN2B7PENWNKW6AXH4DUWRXQQQ3J35GHB43XH5R7BHADKQ, seg_pos={0 0}, upload=438316]: Long tail cancellation due to optimal threshold Dec 26 10:53:33.458939712 (499h24m33.330022758s) [noise=true , address=45.91.116.9:4263 , piece_id=RQFXEENVR2OGDX4O3LTJSXCQ2UZWD7PVFTMX3THJFE56VB5JALVA, seg_pos={0 0}, upload=438316]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 10:53:33.458950723 (499h24m33.330033758s) [noise=true , address=45.91.116.9:4263 , piece_id=RQFXEENVR2OGDX4O3LTJSXCQ2UZWD7PVFTMX3THJFE56VB5JALVA, seg_pos={0 0}, upload=438316]: Long tail cancellation due to optimal threshold Dec 26 10:53:33.458976022 (499h24m33.330059057s) [seg_pos={0 0}, upload=438316]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:53:33.459081743 (499h24m33.330164778s) [upload=438316]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:53:33.459088686 (499h24m33.330171731s) [upload=438316]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:53:33.738965397 (499h24m33.610048432s) [upload=438316]: Done uploading segments... Dec 26 10:53:33.738990455 (499h24m33.61007349s) [upload=438316]: Upload finished. err: <nil> Duration: 30.002919523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X.......XXX......X............X.........X.........] duration: 46.081296735s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 10:53:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 46.081314599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X.......XXX......X............X.........X........] duration: 44.041441459s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 10:52:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 44.041470334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...X.X.XX....XX.......X.X.....X..X....X..X..X.......XX.XX..] duration: 6.723994887s trace: https://jaeger.dev.storj.io/trace/2fbc9b4b4f0ce11a FAIL (2025-12-26 10:52:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: M2TTR2U7H3NLKLTCQ5OJUHQIDTENG534GCZVNIOT7VTWBLYWDNVA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: XFP53PDJUOTUBD463GGAEPPG7GXPOZ5VTDVHNPOCUKIPJG6YR2AA) context deadline exceeded ======LOGS====== Dec 26 10:50:44.579092849 (499h21m44.450175894s) [upload=438280]: Starting upload Dec 26 10:50:44.579219730 (499h21m44.450302775s) [upload=438280]: Done starting upload Dec 26 10:50:44.579326353 (499h21m44.450409398s) [upload=438280]: Uploading segments... Dec 26 10:50:44.579536714 (499h21m44.450619769s) [upload=438280]: Got next segment. Inline: false Dec 26 10:50:44.579580007 (499h21m44.450663052s) [upload=438280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:50:44.579588853 (499h21m44.450671898s) [upload=438280]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:50:44.819698431 (499h21m44.690781486s) [seg_pos={0 0}, upload=438280]: Begin upload segment... Dec 26 10:50:44.819767693 (499h21m44.690850738s) [seg_pos={0 0}, upload=438280]: Done begin upload segment. Dec 26 10:50:44.819819823 (499h21m44.690902858s) [noise=true , address=77.90.31.18:4808 , piece_id=M2TTR2U7H3NLKLTCQ5OJUHQIDTENG534GCZVNIOT7VTWBLYWDNVA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.819841794 (499h21m44.690924839s) [noise=true , address=145.79.227.48:4303 , piece_id=2PXY2XCUKWHZFPG7EMNCI6OMDNUTNHS5JNQHYGZ4QYSJXACOS7YQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.820566344 (499h21m44.691649389s) [noise=true , address=45.91.116.13:4153 , piece_id=SAGCUWVQ56AM64MXQUZFL43M4JFYQIOKNQGQLNHUAJKIX6EPFRVQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.820768188 (499h21m44.691851223s) [noise=true , address=77.90.31.16:4735 , piece_id=B5PDNAXQEGX5KNNQ7JBBVWRSJBWHNPTHXVR4L3TS6UBRKEGKMZAA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.821916605 (499h21m44.69299965s) [noise=true , address=145.79.227.19:4077 , piece_id=IU4XWEC47PGKN55HCSDLX3ZDRZPZNNWVSN3WCLNH6XMA2XKMGADQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.822048757 (499h21m44.693131792s) [noise=true , address=145.79.227.29:4943 , piece_id=EBSEU7ZGARQ7CBITE4FC2FZQPNMF6B2LSKSOPA5L7FQVY5L7TV4Q, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.822932529 (499h21m44.694015574s) [noise=true , address=45.91.116.9:4261 , piece_id=EVFDLP7A32MVYOJ42BKA3MGOIO36QB27QIWZRXKXRJ3KASCKFO7A, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.823235797 (499h21m44.694318832s) [noise=true , address=145.79.227.24:4583 , piece_id=SLX5TAMMWUDBQT4OGGJR5PDVA7LVB7S7T5557SMHKRAZE3UOS3KQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.824492229 (499h21m44.695575264s) [noise=true , address=77.90.31.12:4590 , piece_id=XFP53PDJUOTUBD463GGAEPPG7GXPOZ5VTDVHNPOCUKIPJG6YR2AA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.824607769 (499h21m44.695690804s) [noise=true , address=45.91.116.10:4084 , piece_id=RPKPCRMLYVJHB7JBMILI2KMWIECGWACFGRQTIJIRAUFDDXTQDW5A, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.825770233 (499h21m44.696853268s) [noise=true , address=145.79.227.5:4223 , piece_id=Y3YXXNH2JI5LMVMXW6B5BY4N4SHL47TYHFGYG3ECYI2O35LHEDSA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.826202045 (499h21m44.69728508s) [noise=true , address=45.91.116.7:4012 , piece_id=SARRHQG5CVZOHIF3F4QROGDA7QUALTMM47WAML5WD5HRT6Q2V2UA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.827093603 (499h21m44.698176648s) [noise=true , address=145.79.227.51:4409 , piece_id=BM5F6U7MWNFVT4BRRG7QL56D4RPVWS3KA6H6CYPE6V3L7EXOVBPQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.827614245 (499h21m44.69869728s) [noise=true , address=145.79.227.52:4446 , piece_id=QEEATKZARR4BOSHJYRNOWQQDRH56D5HMJP5XB7QJYT6OIQ7T4VCQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.828785975 (499h21m44.69986903s) [noise=true , address=145.79.227.36:4518 , piece_id=BIHDFRTHKNYGPDNNPWOBDHJS5ZEMO2ZHMQ35LS4N6KU7Q227IMOA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.832867458 (499h21m44.703950493s) [noise=true , address=145.79.227.42:4122 , piece_id=IFV5K7CAHMLWLC5B545ZVEWJZ35OS3EBXXX2GWFVSBCPKXVJZDQA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:46.452944573 (499h21m46.324027608s) [upload=438280]: Next returned nil segment Dec 26 10:50:46.452957327 (499h21m46.324040362s) [upload=438280]: Waiting for error group managing segments... Dec 26 10:50:46.837084108 (499h21m46.708167143s) [noise=true , address=145.79.227.5:4223 , piece_id=Y3YXXNH2JI5LMVMXW6B5BY4N4SHL47TYHFGYG3ECYI2O35LHEDSA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.838147752 (499h21m46.709230797s) [noise=true , address=145.79.227.29:4943 , piece_id=EBSEU7ZGARQ7CBITE4FC2FZQPNMF6B2LSKSOPA5L7FQVY5L7TV4Q, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.839243189 (499h21m46.710326224s) [noise=true , address=145.79.227.24:4583 , piece_id=SLX5TAMMWUDBQT4OGGJR5PDVA7LVB7S7T5557SMHKRAZE3UOS3KQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.840349525 (499h21m46.71143256s) [noise=true , address=145.79.227.19:4077 , piece_id=IU4XWEC47PGKN55HCSDLX3ZDRZPZNNWVSN3WCLNH6XMA2XKMGADQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.842780123 (499h21m46.713863158s) [noise=true , address=145.79.227.42:4122 , piece_id=IFV5K7CAHMLWLC5B545ZVEWJZ35OS3EBXXX2GWFVSBCPKXVJZDQA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.843977424 (499h21m46.715060459s) [noise=true , address=145.79.227.36:4518 , piece_id=BIHDFRTHKNYGPDNNPWOBDHJS5ZEMO2ZHMQ35LS4N6KU7Q227IMOA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.854742752 (499h21m46.725825787s) [noise=true , address=145.79.227.51:4409 , piece_id=BM5F6U7MWNFVT4BRRG7QL56D4RPVWS3KA6H6CYPE6V3L7EXOVBPQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.855001625 (499h21m46.72608466s) [noise=true , address=145.79.227.48:4303 , piece_id=2PXY2XCUKWHZFPG7EMNCI6OMDNUTNHS5JNQHYGZ4QYSJXACOS7YQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.855241892 (499h21m46.726324928s) [noise=true , address=145.79.227.52:4446 , piece_id=QEEATKZARR4BOSHJYRNOWQQDRH56D5HMJP5XB7QJYT6OIQ7T4VCQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.102211814 (499h21m46.973294859s) [noise=true , address=45.91.116.9:4261 , piece_id=EVFDLP7A32MVYOJ42BKA3MGOIO36QB27QIWZRXKXRJ3KASCKFO7A, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.162125109 (499h21m47.033208144s) [noise=true , address=45.91.116.10:4084 , piece_id=RPKPCRMLYVJHB7JBMILI2KMWIECGWACFGRQTIJIRAUFDDXTQDW5A, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.226241106 (499h21m47.097324151s) [noise=true , address=77.90.31.18:4808 , piece_id=M2TTR2U7H3NLKLTCQ5OJUHQIDTENG534GCZVNIOT7VTWBLYWDNVA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.262981767 (499h21m47.134064802s) [noise=true , address=77.90.31.12:4590 , piece_id=XFP53PDJUOTUBD463GGAEPPG7GXPOZ5VTDVHNPOCUKIPJG6YR2AA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.263002306 (499h21m47.134085341s) [seg_pos={0 0}, upload=438280]: Segment reached optimal threshold of 13 pieces. Dec 26 10:50:47.263207517 (499h21m47.134290552s) [noise=true , address=77.90.31.16:4735 , piece_id=B5PDNAXQEGX5KNNQ7JBBVWRSJBWHNPTHXVR4L3TS6UBRKEGKMZAA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 10:50:47.263214029 (499h21m47.134297064s) [noise=true , address=77.90.31.16:4735 , piece_id=B5PDNAXQEGX5KNNQ7JBBVWRSJBWHNPTHXVR4L3TS6UBRKEGKMZAA, seg_pos={0 0}, upload=438280]: Long tail cancellation due to optimal threshold Dec 26 10:50:47.263367671 (499h21m47.134450706s) [noise=true , address=45.91.116.7:4012 , piece_id=SARRHQG5CVZOHIF3F4QROGDA7QUALTMM47WAML5WD5HRT6Q2V2UA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:44246->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:44246->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44246->45.91.116.7:4012: use of closed network connection Dec 26 10:50:47.263384113 (499h21m47.134467148s) [noise=true , address=45.91.116.7:4012 , piece_id=SARRHQG5CVZOHIF3F4QROGDA7QUALTMM47WAML5WD5HRT6Q2V2UA, seg_pos={0 0}, upload=438280]: Long tail cancellation due to optimal threshold Dec 26 10:50:47.263416936 (499h21m47.134499971s) [noise=true , address=45.91.116.13:4153 , piece_id=SAGCUWVQ56AM64MXQUZFL43M4JFYQIOKNQGQLNHUAJKIX6EPFRVQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 10:50:47.263425882 (499h21m47.134508918s) [noise=true , address=45.91.116.13:4153 , piece_id=SAGCUWVQ56AM64MXQUZFL43M4JFYQIOKNQGQLNHUAJKIX6EPFRVQ, seg_pos={0 0}, upload=438280]: Long tail cancellation due to optimal threshold Dec 26 10:50:47.263448185 (499h21m47.13453122s) [seg_pos={0 0}, upload=438280]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:50:47.263568464 (499h21m47.134651509s) [upload=438280]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:50:47.263577230 (499h21m47.134660275s) [upload=438280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:50:47.545625167 (499h21m47.416708202s) [upload=438280]: Done uploading segments... Dec 26 10:50:47.545636929 (499h21m47.416719964s) [upload=438280]: Upload finished. err: <nil> Duration: 6.724023581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX...X.X.XX....XX.......X.X.....X..X....X..X..X.......XX.XX.] duration: 5.763798953s trace: https://jaeger.dev.storj.io/trace/2fbc9b4b4f0ce11a FAIL (2025-12-26 10:52:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: M2TTR2U7H3NLKLTCQ5OJUHQIDTENG534GCZVNIOT7VTWBLYWDNVA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: XFP53PDJUOTUBD463GGAEPPG7GXPOZ5VTDVHNPOCUKIPJG6YR2AA) context deadline exceeded ======LOGS====== Dec 26 10:50:44.579092849 (499h21m44.450175894s) [upload=438280]: Starting upload Dec 26 10:50:44.579219730 (499h21m44.450302775s) [upload=438280]: Done starting upload Dec 26 10:50:44.579326353 (499h21m44.450409398s) [upload=438280]: Uploading segments... Dec 26 10:50:44.579536714 (499h21m44.450619769s) [upload=438280]: Got next segment. Inline: false Dec 26 10:50:44.579580007 (499h21m44.450663052s) [upload=438280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:50:44.579588853 (499h21m44.450671898s) [upload=438280]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:50:44.819698431 (499h21m44.690781486s) [seg_pos={0 0}, upload=438280]: Begin upload segment... Dec 26 10:50:44.819767693 (499h21m44.690850738s) [seg_pos={0 0}, upload=438280]: Done begin upload segment. Dec 26 10:50:44.819819823 (499h21m44.690902858s) [noise=true , address=77.90.31.18:4808 , piece_id=M2TTR2U7H3NLKLTCQ5OJUHQIDTENG534GCZVNIOT7VTWBLYWDNVA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.819841794 (499h21m44.690924839s) [noise=true , address=145.79.227.48:4303 , piece_id=2PXY2XCUKWHZFPG7EMNCI6OMDNUTNHS5JNQHYGZ4QYSJXACOS7YQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.820566344 (499h21m44.691649389s) [noise=true , address=45.91.116.13:4153 , piece_id=SAGCUWVQ56AM64MXQUZFL43M4JFYQIOKNQGQLNHUAJKIX6EPFRVQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.820768188 (499h21m44.691851223s) [noise=true , address=77.90.31.16:4735 , piece_id=B5PDNAXQEGX5KNNQ7JBBVWRSJBWHNPTHXVR4L3TS6UBRKEGKMZAA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.821916605 (499h21m44.69299965s) [noise=true , address=145.79.227.19:4077 , piece_id=IU4XWEC47PGKN55HCSDLX3ZDRZPZNNWVSN3WCLNH6XMA2XKMGADQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.822048757 (499h21m44.693131792s) [noise=true , address=145.79.227.29:4943 , piece_id=EBSEU7ZGARQ7CBITE4FC2FZQPNMF6B2LSKSOPA5L7FQVY5L7TV4Q, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.822932529 (499h21m44.694015574s) [noise=true , address=45.91.116.9:4261 , piece_id=EVFDLP7A32MVYOJ42BKA3MGOIO36QB27QIWZRXKXRJ3KASCKFO7A, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.823235797 (499h21m44.694318832s) [noise=true , address=145.79.227.24:4583 , piece_id=SLX5TAMMWUDBQT4OGGJR5PDVA7LVB7S7T5557SMHKRAZE3UOS3KQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.824492229 (499h21m44.695575264s) [noise=true , address=77.90.31.12:4590 , piece_id=XFP53PDJUOTUBD463GGAEPPG7GXPOZ5VTDVHNPOCUKIPJG6YR2AA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.824607769 (499h21m44.695690804s) [noise=true , address=45.91.116.10:4084 , piece_id=RPKPCRMLYVJHB7JBMILI2KMWIECGWACFGRQTIJIRAUFDDXTQDW5A, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.825770233 (499h21m44.696853268s) [noise=true , address=145.79.227.5:4223 , piece_id=Y3YXXNH2JI5LMVMXW6B5BY4N4SHL47TYHFGYG3ECYI2O35LHEDSA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.826202045 (499h21m44.69728508s) [noise=true , address=45.91.116.7:4012 , piece_id=SARRHQG5CVZOHIF3F4QROGDA7QUALTMM47WAML5WD5HRT6Q2V2UA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.827093603 (499h21m44.698176648s) [noise=true , address=145.79.227.51:4409 , piece_id=BM5F6U7MWNFVT4BRRG7QL56D4RPVWS3KA6H6CYPE6V3L7EXOVBPQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.827614245 (499h21m44.69869728s) [noise=true , address=145.79.227.52:4446 , piece_id=QEEATKZARR4BOSHJYRNOWQQDRH56D5HMJP5XB7QJYT6OIQ7T4VCQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.828785975 (499h21m44.69986903s) [noise=true , address=145.79.227.36:4518 , piece_id=BIHDFRTHKNYGPDNNPWOBDHJS5ZEMO2ZHMQ35LS4N6KU7Q227IMOA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.832867458 (499h21m44.703950493s) [noise=true , address=145.79.227.42:4122 , piece_id=IFV5K7CAHMLWLC5B545ZVEWJZ35OS3EBXXX2GWFVSBCPKXVJZDQA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:46.452944573 (499h21m46.324027608s) [upload=438280]: Next returned nil segment Dec 26 10:50:46.452957327 (499h21m46.324040362s) [upload=438280]: Waiting for error group managing segments... Dec 26 10:50:46.837084108 (499h21m46.708167143s) [noise=true , address=145.79.227.5:4223 , piece_id=Y3YXXNH2JI5LMVMXW6B5BY4N4SHL47TYHFGYG3ECYI2O35LHEDSA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.838147752 (499h21m46.709230797s) [noise=true , address=145.79.227.29:4943 , piece_id=EBSEU7ZGARQ7CBITE4FC2FZQPNMF6B2LSKSOPA5L7FQVY5L7TV4Q, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.839243189 (499h21m46.710326224s) [noise=true , address=145.79.227.24:4583 , piece_id=SLX5TAMMWUDBQT4OGGJR5PDVA7LVB7S7T5557SMHKRAZE3UOS3KQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.840349525 (499h21m46.71143256s) [noise=true , address=145.79.227.19:4077 , piece_id=IU4XWEC47PGKN55HCSDLX3ZDRZPZNNWVSN3WCLNH6XMA2XKMGADQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.842780123 (499h21m46.713863158s) [noise=true , address=145.79.227.42:4122 , piece_id=IFV5K7CAHMLWLC5B545ZVEWJZ35OS3EBXXX2GWFVSBCPKXVJZDQA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.843977424 (499h21m46.715060459s) [noise=true , address=145.79.227.36:4518 , piece_id=BIHDFRTHKNYGPDNNPWOBDHJS5ZEMO2ZHMQ35LS4N6KU7Q227IMOA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.854742752 (499h21m46.725825787s) [noise=true , address=145.79.227.51:4409 , piece_id=BM5F6U7MWNFVT4BRRG7QL56D4RPVWS3KA6H6CYPE6V3L7EXOVBPQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.855001625 (499h21m46.72608466s) [noise=true , address=145.79.227.48:4303 , piece_id=2PXY2XCUKWHZFPG7EMNCI6OMDNUTNHS5JNQHYGZ4QYSJXACOS7YQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.855241892 (499h21m46.726324928s) [noise=true , address=145.79.227.52:4446 , piece_id=QEEATKZARR4BOSHJYRNOWQQDRH56D5HMJP5XB7QJYT6OIQ7T4VCQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.102211814 (499h21m46.973294859s) [noise=true , address=45.91.116.9:4261 , piece_id=EVFDLP7A32MVYOJ42BKA3MGOIO36QB27QIWZRXKXRJ3KASCKFO7A, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.162125109 (499h21m47.033208144s) [noise=true , address=45.91.116.10:4084 , piece_id=RPKPCRMLYVJHB7JBMILI2KMWIECGWACFGRQTIJIRAUFDDXTQDW5A, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.226241106 (499h21m47.097324151s) [noise=true , address=77.90.31.18:4808 , piece_id=M2TTR2U7H3NLKLTCQ5OJUHQIDTENG534GCZVNIOT7VTWBLYWDNVA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.262981767 (499h21m47.134064802s) [noise=true , address=77.90.31.12:4590 , piece_id=XFP53PDJUOTUBD463GGAEPPG7GXPOZ5VTDVHNPOCUKIPJG6YR2AA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.263002306 (499h21m47.134085341s) [seg_pos={0 0}, upload=438280]: Segment reached optimal threshold of 13 pieces. Dec 26 10:50:47.263207517 (499h21m47.134290552s) [noise=true , address=77.90.31.16:4735 , piece_id=B5PDNAXQEGX5KNNQ7JBBVWRSJBWHNPTHXVR4L3TS6UBRKEGKMZAA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 10:50:47.263214029 (499h21m47.134297064s) [noise=true , address=77.90.31.16:4735 , piece_id=B5PDNAXQEGX5KNNQ7JBBVWRSJBWHNPTHXVR4L3TS6UBRKEGKMZAA, seg_pos={0 0}, upload=438280]: Long tail cancellation due to optimal threshold Dec 26 10:50:47.263367671 (499h21m47.134450706s) [noise=true , address=45.91.116.7:4012 , piece_id=SARRHQG5CVZOHIF3F4QROGDA7QUALTMM47WAML5WD5HRT6Q2V2UA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:44246->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:44246->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44246->45.91.116.7:4012: use of closed network connection Dec 26 10:50:47.263384113 (499h21m47.134467148s) [noise=true , address=45.91.116.7:4012 , piece_id=SARRHQG5CVZOHIF3F4QROGDA7QUALTMM47WAML5WD5HRT6Q2V2UA, seg_pos={0 0}, upload=438280]: Long tail cancellation due to optimal threshold Dec 26 10:50:47.263416936 (499h21m47.134499971s) [noise=true , address=45.91.116.13:4153 , piece_id=SAGCUWVQ56AM64MXQUZFL43M4JFYQIOKNQGQLNHUAJKIX6EPFRVQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 10:50:47.263425882 (499h21m47.134508918s) [noise=true , address=45.91.116.13:4153 , piece_id=SAGCUWVQ56AM64MXQUZFL43M4JFYQIOKNQGQLNHUAJKIX6EPFRVQ, seg_pos={0 0}, upload=438280]: Long tail cancellation due to optimal threshold Dec 26 10:50:47.263448185 (499h21m47.13453122s) [seg_pos={0 0}, upload=438280]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:50:47.263568464 (499h21m47.134651509s) [upload=438280]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:50:47.263577230 (499h21m47.134660275s) [upload=438280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:50:47.545625167 (499h21m47.416708202s) [upload=438280]: Done uploading segments... Dec 26 10:50:47.545636929 (499h21m47.416719964s) [upload=438280]: Upload finished. err: <nil> Duration: 5.763832878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X.......XXX......X............X.........X.......] duration: 38.899114656s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 10:51:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 38.899129254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX...X.X.XX....XX.......X.X.....X..X....X..X..X.......XX.XX] duration: 30.002970408s trace: https://jaeger.dev.storj.io/trace/2fbc9b4b4f0ce11a FAIL (2025-12-26 10:51:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: M2TTR2U7H3NLKLTCQ5OJUHQIDTENG534GCZVNIOT7VTWBLYWDNVA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: XFP53PDJUOTUBD463GGAEPPG7GXPOZ5VTDVHNPOCUKIPJG6YR2AA) context deadline exceeded ======LOGS====== Dec 26 10:50:44.579092849 (499h21m44.450175894s) [upload=438280]: Starting upload Dec 26 10:50:44.579219730 (499h21m44.450302775s) [upload=438280]: Done starting upload Dec 26 10:50:44.579326353 (499h21m44.450409398s) [upload=438280]: Uploading segments... Dec 26 10:50:44.579536714 (499h21m44.450619769s) [upload=438280]: Got next segment. Inline: false Dec 26 10:50:44.579580007 (499h21m44.450663052s) [upload=438280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:50:44.579588853 (499h21m44.450671898s) [upload=438280]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:50:44.819698431 (499h21m44.690781486s) [seg_pos={0 0}, upload=438280]: Begin upload segment... Dec 26 10:50:44.819767693 (499h21m44.690850738s) [seg_pos={0 0}, upload=438280]: Done begin upload segment. Dec 26 10:50:44.819819823 (499h21m44.690902858s) [noise=true , address=77.90.31.18:4808 , piece_id=M2TTR2U7H3NLKLTCQ5OJUHQIDTENG534GCZVNIOT7VTWBLYWDNVA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.819841794 (499h21m44.690924839s) [noise=true , address=145.79.227.48:4303 , piece_id=2PXY2XCUKWHZFPG7EMNCI6OMDNUTNHS5JNQHYGZ4QYSJXACOS7YQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.820566344 (499h21m44.691649389s) [noise=true , address=45.91.116.13:4153 , piece_id=SAGCUWVQ56AM64MXQUZFL43M4JFYQIOKNQGQLNHUAJKIX6EPFRVQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.820768188 (499h21m44.691851223s) [noise=true , address=77.90.31.16:4735 , piece_id=B5PDNAXQEGX5KNNQ7JBBVWRSJBWHNPTHXVR4L3TS6UBRKEGKMZAA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.821916605 (499h21m44.69299965s) [noise=true , address=145.79.227.19:4077 , piece_id=IU4XWEC47PGKN55HCSDLX3ZDRZPZNNWVSN3WCLNH6XMA2XKMGADQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.822048757 (499h21m44.693131792s) [noise=true , address=145.79.227.29:4943 , piece_id=EBSEU7ZGARQ7CBITE4FC2FZQPNMF6B2LSKSOPA5L7FQVY5L7TV4Q, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.822932529 (499h21m44.694015574s) [noise=true , address=45.91.116.9:4261 , piece_id=EVFDLP7A32MVYOJ42BKA3MGOIO36QB27QIWZRXKXRJ3KASCKFO7A, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.823235797 (499h21m44.694318832s) [noise=true , address=145.79.227.24:4583 , piece_id=SLX5TAMMWUDBQT4OGGJR5PDVA7LVB7S7T5557SMHKRAZE3UOS3KQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.824492229 (499h21m44.695575264s) [noise=true , address=77.90.31.12:4590 , piece_id=XFP53PDJUOTUBD463GGAEPPG7GXPOZ5VTDVHNPOCUKIPJG6YR2AA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.824607769 (499h21m44.695690804s) [noise=true , address=45.91.116.10:4084 , piece_id=RPKPCRMLYVJHB7JBMILI2KMWIECGWACFGRQTIJIRAUFDDXTQDW5A, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.825770233 (499h21m44.696853268s) [noise=true , address=145.79.227.5:4223 , piece_id=Y3YXXNH2JI5LMVMXW6B5BY4N4SHL47TYHFGYG3ECYI2O35LHEDSA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.826202045 (499h21m44.69728508s) [noise=true , address=45.91.116.7:4012 , piece_id=SARRHQG5CVZOHIF3F4QROGDA7QUALTMM47WAML5WD5HRT6Q2V2UA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.827093603 (499h21m44.698176648s) [noise=true , address=145.79.227.51:4409 , piece_id=BM5F6U7MWNFVT4BRRG7QL56D4RPVWS3KA6H6CYPE6V3L7EXOVBPQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.827614245 (499h21m44.69869728s) [noise=true , address=145.79.227.52:4446 , piece_id=QEEATKZARR4BOSHJYRNOWQQDRH56D5HMJP5XB7QJYT6OIQ7T4VCQ, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.828785975 (499h21m44.69986903s) [noise=true , address=145.79.227.36:4518 , piece_id=BIHDFRTHKNYGPDNNPWOBDHJS5ZEMO2ZHMQ35LS4N6KU7Q227IMOA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:44.832867458 (499h21m44.703950493s) [noise=true , address=145.79.227.42:4122 , piece_id=IFV5K7CAHMLWLC5B545ZVEWJZ35OS3EBXXX2GWFVSBCPKXVJZDQA, seg_pos={0 0}, upload=438280]: Uploading piece... Dec 26 10:50:46.452944573 (499h21m46.324027608s) [upload=438280]: Next returned nil segment Dec 26 10:50:46.452957327 (499h21m46.324040362s) [upload=438280]: Waiting for error group managing segments... Dec 26 10:50:46.837084108 (499h21m46.708167143s) [noise=true , address=145.79.227.5:4223 , piece_id=Y3YXXNH2JI5LMVMXW6B5BY4N4SHL47TYHFGYG3ECYI2O35LHEDSA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.838147752 (499h21m46.709230797s) [noise=true , address=145.79.227.29:4943 , piece_id=EBSEU7ZGARQ7CBITE4FC2FZQPNMF6B2LSKSOPA5L7FQVY5L7TV4Q, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.839243189 (499h21m46.710326224s) [noise=true , address=145.79.227.24:4583 , piece_id=SLX5TAMMWUDBQT4OGGJR5PDVA7LVB7S7T5557SMHKRAZE3UOS3KQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.840349525 (499h21m46.71143256s) [noise=true , address=145.79.227.19:4077 , piece_id=IU4XWEC47PGKN55HCSDLX3ZDRZPZNNWVSN3WCLNH6XMA2XKMGADQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.842780123 (499h21m46.713863158s) [noise=true , address=145.79.227.42:4122 , piece_id=IFV5K7CAHMLWLC5B545ZVEWJZ35OS3EBXXX2GWFVSBCPKXVJZDQA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.843977424 (499h21m46.715060459s) [noise=true , address=145.79.227.36:4518 , piece_id=BIHDFRTHKNYGPDNNPWOBDHJS5ZEMO2ZHMQ35LS4N6KU7Q227IMOA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.854742752 (499h21m46.725825787s) [noise=true , address=145.79.227.51:4409 , piece_id=BM5F6U7MWNFVT4BRRG7QL56D4RPVWS3KA6H6CYPE6V3L7EXOVBPQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.855001625 (499h21m46.72608466s) [noise=true , address=145.79.227.48:4303 , piece_id=2PXY2XCUKWHZFPG7EMNCI6OMDNUTNHS5JNQHYGZ4QYSJXACOS7YQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:46.855241892 (499h21m46.726324928s) [noise=true , address=145.79.227.52:4446 , piece_id=QEEATKZARR4BOSHJYRNOWQQDRH56D5HMJP5XB7QJYT6OIQ7T4VCQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.102211814 (499h21m46.973294859s) [noise=true , address=45.91.116.9:4261 , piece_id=EVFDLP7A32MVYOJ42BKA3MGOIO36QB27QIWZRXKXRJ3KASCKFO7A, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.162125109 (499h21m47.033208144s) [noise=true , address=45.91.116.10:4084 , piece_id=RPKPCRMLYVJHB7JBMILI2KMWIECGWACFGRQTIJIRAUFDDXTQDW5A, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.226241106 (499h21m47.097324151s) [noise=true , address=77.90.31.18:4808 , piece_id=M2TTR2U7H3NLKLTCQ5OJUHQIDTENG534GCZVNIOT7VTWBLYWDNVA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.262981767 (499h21m47.134064802s) [noise=true , address=77.90.31.12:4590 , piece_id=XFP53PDJUOTUBD463GGAEPPG7GXPOZ5VTDVHNPOCUKIPJG6YR2AA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: <nil> Dec 26 10:50:47.263002306 (499h21m47.134085341s) [seg_pos={0 0}, upload=438280]: Segment reached optimal threshold of 13 pieces. Dec 26 10:50:47.263207517 (499h21m47.134290552s) [noise=true , address=77.90.31.16:4735 , piece_id=B5PDNAXQEGX5KNNQ7JBBVWRSJBWHNPTHXVR4L3TS6UBRKEGKMZAA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 10:50:47.263214029 (499h21m47.134297064s) [noise=true , address=77.90.31.16:4735 , piece_id=B5PDNAXQEGX5KNNQ7JBBVWRSJBWHNPTHXVR4L3TS6UBRKEGKMZAA, seg_pos={0 0}, upload=438280]: Long tail cancellation due to optimal threshold Dec 26 10:50:47.263367671 (499h21m47.134450706s) [noise=true , address=45.91.116.7:4012 , piece_id=SARRHQG5CVZOHIF3F4QROGDA7QUALTMM47WAML5WD5HRT6Q2V2UA, seg_pos={0 0}, upload=438280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:44246->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:44246->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44246->45.91.116.7:4012: use of closed network connection Dec 26 10:50:47.263384113 (499h21m47.134467148s) [noise=true , address=45.91.116.7:4012 , piece_id=SARRHQG5CVZOHIF3F4QROGDA7QUALTMM47WAML5WD5HRT6Q2V2UA, seg_pos={0 0}, upload=438280]: Long tail cancellation due to optimal threshold Dec 26 10:50:47.263416936 (499h21m47.134499971s) [noise=true , address=45.91.116.13:4153 , piece_id=SAGCUWVQ56AM64MXQUZFL43M4JFYQIOKNQGQLNHUAJKIX6EPFRVQ, seg_pos={0 0}, upload=438280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 10:50:47.263425882 (499h21m47.134508918s) [noise=true , address=45.91.116.13:4153 , piece_id=SAGCUWVQ56AM64MXQUZFL43M4JFYQIOKNQGQLNHUAJKIX6EPFRVQ, seg_pos={0 0}, upload=438280]: Long tail cancellation due to optimal threshold Dec 26 10:50:47.263448185 (499h21m47.13453122s) [seg_pos={0 0}, upload=438280]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:50:47.263568464 (499h21m47.134651509s) [upload=438280]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:50:47.263577230 (499h21m47.134660275s) [upload=438280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:50:47.545625167 (499h21m47.416708202s) [upload=438280]: Done uploading segments... Dec 26 10:50:47.545636929 (499h21m47.416719964s) [upload=438280]: Upload finished. err: <nil> Duration: 30.002995494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X.......XXX......X............X.........X......] duration: 51.879349464s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 10:50:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 51.879377307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX...X.X.XX....XX.......X.X.....X..X....X..X..X.......XX.X] duration: 30.003908489s trace: https://jaeger.dev.storj.io/trace/5af72d2aac7c9c6e FAIL (2025-12-26 10:50:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: ESEI2TFJKIKCUZFUVMB4O7TENB57IKGNBTHJID3J3O4CM4PTZNUQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: GK2CUODKYSUMU2ILZQK3U3MDYDUTLM27EL3V66IGMWJFOJ2OVGEQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: LPINEL6ZWTQ67LHY5PUZ4AW34KBSTGTJLWS5MTZMGK2SPJLSH6NQ) context deadline exceeded ======LOGS====== 4 (499h21m1.239696409s) [upload=438269]: Uploading segments... Dec 26 10:50:01.368787426 (499h21m1.239870471s) [upload=438269]: Got next segment. Inline: false Dec 26 10:50:01.368841830 (499h21m1.239924865s) [upload=438269]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:50:01.368848182 (499h21m1.239931217s) [upload=438269]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:50:01.615849332 (499h21m1.486932367s) [seg_pos={0 0}, upload=438269]: Begin upload segment... Dec 26 10:50:01.615914166 (499h21m1.486997211s) [seg_pos={0 0}, upload=438269]: Done begin upload segment. Dec 26 10:50:01.615951867 (499h21m1.487034912s) [noise=true , address=145.79.227.2:4115 , piece_id=RJFUDY4BVEBHIYMJNEU6UVTFJL4TZ35LXQBUR7B2ZQJEZEP435BQ, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.616469072 (499h21m1.487552117s) [noise=true , address=45.91.116.6:4975 , piece_id=RHDSV67WAR2ZYI27C4OR2AV7LVPKM56BNI4ON6NPBW5L3O3VXUMA, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.616895454 (499h21m1.487978499s) [noise=true , address=45.91.116.9:4261 , piece_id=GK2CUODKYSUMU2ILZQK3U3MDYDUTLM27EL3V66IGMWJFOJ2OVGEQ, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.618351907 (499h21m1.489434953s) [noise=true , address=145.79.227.38:4979 , piece_id=IBRM7EOIN7FDV5GZ5JJTSOPBBHRLS4JQYQMSEDJZTFXGMIBFGQ6A, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.619004941 (499h21m1.490087986s) [noise=true , address=145.79.227.22:4511 , piece_id=BCM5WNTT3QKK2VHMZRUY2T2UUSPOXCHEKWQHTZOK3Y574PA7PIIQ, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.620417931 (499h21m1.491500976s) [noise=true , address=145.79.227.7:4293 , piece_id=5DTTC5WE3X6EZV2RCU324OWRSNFIJFJDD2BMKYJEXQFSH7JWTXEQ, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.621178079 (499h21m1.492261134s) [noise=true , address=145.79.227.49:4339 , piece_id=V6L5QQH3WQYBCEKJG4TDACVIIR5F4XILIO2JAYQHAWA2NDJTDNHA, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.621658594 (499h21m1.492741649s) [noise=true , address=77.90.31.10:4518 , piece_id=LPINEL6ZWTQ67LHY5PUZ4AW34KBSTGTJLWS5MTZMGK2SPJLSH6NQ, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.623006471 (499h21m1.494089516s) [noise=true , address=145.79.227.11:4730 , piece_id=OEKUGGI5OQL2XFCWF33WALBJQ3VPLF57OBOKE4Q3NSGFNBCZBEYQ, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.623128833 (499h21m1.494211878s) [noise=true , address=45.91.116.10:4082 , piece_id=FTXURBHWHFP7LYSCPEBFZQ3CHC56E67URWYWAZZTSVYL4KOM3KYA, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.623380071 (499h21m1.494463147s) [noise=true , address=145.79.227.17:4971 , piece_id=RL6LM54525GMIVCCJDUQSRLGLBIJWIYR76GWL23L26FIPLW2SLKA, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.623853293 (499h21m1.494936338s) [noise=true , address=45.91.116.7:4012 , piece_id=7R74FRKZEZYUYEX74HIECF4MGN3JYF2DDKH37PHROXIZJTZZ6A7A, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.624091306 (499h21m1.495174351s) [noise=true , address=45.91.116.4:4937 , piece_id=73ZMZZIMHM3R7EL6YYDQKWRURPCB2XGGDWYM4VXESAZNXGXX4DEQ, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.624532205 (499h21m1.49561525s) [noise=true , address=45.91.116.13:4153 , piece_id=SNSUD5WA4KJQT67HI52DHJFZBGOAXO5APGLN6D3SAOTTCUFXXGVQ, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.625213232 (499h21m1.496296278s) [noise=true , address=145.79.227.36:4519 , piece_id=D6ZQPOVC2TGMSWGI6EDTRSS4VUVX4AX6PNE65DRQLKXSUBWDDMPA, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:01.626273060 (499h21m1.497356115s) [noise=true , address=45.91.116.12:4225 , piece_id=ESEI2TFJKIKCUZFUVMB4O7TENB57IKGNBTHJID3J3O4CM4PTZNUQ, seg_pos={0 0}, upload=438269]: Uploading piece... Dec 26 10:50:03.397507875 (499h21m3.26859093s) [upload=438269]: Next returned nil segment Dec 26 10:50:03.397602345 (499h21m3.26868539s) [upload=438269]: Waiting for error group managing segments... Dec 26 10:50:03.697247489 (499h21m3.568330534s) [noise=true , address=145.79.227.2:4115 , piece_id=RJFUDY4BVEBHIYMJNEU6UVTFJL4TZ35LXQBUR7B2ZQJEZEP435BQ, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:03.698754949 (499h21m3.569837994s) [noise=true , address=145.79.227.17:4971 , piece_id=RL6LM54525GMIVCCJDUQSRLGLBIJWIYR76GWL23L26FIPLW2SLKA, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:03.698788893 (499h21m3.569871938s) [noise=true , address=145.79.227.36:4519 , piece_id=D6ZQPOVC2TGMSWGI6EDTRSS4VUVX4AX6PNE65DRQLKXSUBWDDMPA, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:03.702235717 (499h21m3.573318762s) [noise=true , address=145.79.227.49:4339 , piece_id=V6L5QQH3WQYBCEKJG4TDACVIIR5F4XILIO2JAYQHAWA2NDJTDNHA, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:03.707764475 (499h21m3.57884752s) [noise=true , address=145.79.227.7:4293 , piece_id=5DTTC5WE3X6EZV2RCU324OWRSNFIJFJDD2BMKYJEXQFSH7JWTXEQ, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:03.708086047 (499h21m3.579169102s) [noise=true , address=145.79.227.38:4979 , piece_id=IBRM7EOIN7FDV5GZ5JJTSOPBBHRLS4JQYQMSEDJZTFXGMIBFGQ6A, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:03.716429765 (499h21m3.58751281s) [noise=true , address=145.79.227.11:4730 , piece_id=OEKUGGI5OQL2XFCWF33WALBJQ3VPLF57OBOKE4Q3NSGFNBCZBEYQ, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:03.728525126 (499h21m3.599608171s) [noise=true , address=145.79.227.22:4511 , piece_id=BCM5WNTT3QKK2VHMZRUY2T2UUSPOXCHEKWQHTZOK3Y574PA7PIIQ, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:03.910945976 (499h21m3.782029021s) [noise=true , address=45.91.116.12:4225 , piece_id=ESEI2TFJKIKCUZFUVMB4O7TENB57IKGNBTHJID3J3O4CM4PTZNUQ, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:03.970208141 (499h21m3.841291176s) [noise=true , address=45.91.116.9:4261 , piece_id=GK2CUODKYSUMU2ILZQK3U3MDYDUTLM27EL3V66IGMWJFOJ2OVGEQ, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:04.013460209 (499h21m3.884543255s) [noise=true , address=45.91.116.10:4082 , piece_id=FTXURBHWHFP7LYSCPEBFZQ3CHC56E67URWYWAZZTSVYL4KOM3KYA, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:04.013847607 (499h21m3.884930652s) [noise=true , address=77.90.31.10:4518 , piece_id=LPINEL6ZWTQ67LHY5PUZ4AW34KBSTGTJLWS5MTZMGK2SPJLSH6NQ, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:04.016146985 (499h21m3.88723004s) [noise=true , address=45.91.116.13:4153 , piece_id=SNSUD5WA4KJQT67HI52DHJFZBGOAXO5APGLN6D3SAOTTCUFXXGVQ, seg_pos={0 0}, upload=438269]: Done uploading piece. err: <nil> Dec 26 10:50:04.016189917 (499h21m3.887272962s) [seg_pos={0 0}, upload=438269]: Segment reached optimal threshold of 13 pieces. Dec 26 10:50:04.016671053 (499h21m3.887754108s) [noise=true , address=45.91.116.6:4975 , piece_id=RHDSV67WAR2ZYI27C4OR2AV7LVPKM56BNI4ON6NPBW5L3O3VXUMA, seg_pos={0 0}, upload=438269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:58282->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:58282->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58282->45.91.116.6:4975: use of closed network connection Dec 26 10:50:04.016689328 (499h21m3.887772374s) [noise=true , address=45.91.116.6:4975 , piece_id=RHDSV67WAR2ZYI27C4OR2AV7LVPKM56BNI4ON6NPBW5L3O3VXUMA, seg_pos={0 0}, upload=438269]: Long tail cancellation due to optimal threshold Dec 26 10:50:04.017261257 (499h21m3.888344292s) [noise=true , address=45.91.116.7:4012 , piece_id=7R74FRKZEZYUYEX74HIECF4MGN3JYF2DDKH37PHROXIZJTZZ6A7A, seg_pos={0 0}, upload=438269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:43318->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:43318->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->45.91.116.7:4012: use of closed network connection Dec 26 10:50:04.017275594 (499h21m3.888358639s) [noise=true , address=45.91.116.7:4012 , piece_id=7R74FRKZEZYUYEX74HIECF4MGN3JYF2DDKH37PHROXIZJTZZ6A7A, seg_pos={0 0}, upload=438269]: Long tail cancellation due to optimal threshold Dec 26 10:50:04.017360356 (499h21m3.888443401s) [noise=true , address=45.91.116.4:4937 , piece_id=73ZMZZIMHM3R7EL6YYDQKWRURPCB2XGGDWYM4VXESAZNXGXX4DEQ, seg_pos={0 0}, upload=438269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 10:50:04.017382308 (499h21m3.888465353s) [noise=true , address=45.91.116.4:4937 , piece_id=73ZMZZIMHM3R7EL6YYDQKWRURPCB2XGGDWYM4VXESAZNXGXX4DEQ, seg_pos={0 0}, upload=438269]: Long tail cancellation due to optimal threshold Dec 26 10:50:04.017413817 (499h21m3.888496852s) [seg_pos={0 0}, upload=438269]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:50:04.017504801 (499h21m3.888587846s) [upload=438269]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:50:04.017511744 (499h21m3.888594789s) [upload=438269]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:50:04.297761065 (499h21m4.16884411s) [upload=438269]: Done uploading segments... Dec 26 10:50:04.297786073 (499h21m4.168869118s) [upload=438269]: Upload finished. err: <nil> Duration: 30.003932675s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X.......XXX......X............X.........X.....] duration: 47.3720292s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 10:49:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 47.372067742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..XX...X.X.XX....XX.......X.X.....X..X....X..X..X.......XX.] duration: 6.046326524s trace: https://jaeger.dev.storj.io/trace/483849bb45a54e20 FAIL (2025-12-26 10:49:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 10:48:29.228035124 (499h19m29.099118169s) [upload=438249]: Starting upload Dec 26 10:48:29.228196011 (499h19m29.099279066s) [upload=438249]: Done starting upload Dec 26 10:48:29.228254632 (499h19m29.099337667s) [upload=438249]: Uploading segments... Dec 26 10:48:29.228570444 (499h19m29.099653479s) [upload=438249]: Got next segment. Inline: false Dec 26 10:48:29.228613005 (499h19m29.09969605s) [upload=438249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:48:29.228618926 (499h19m29.099701971s) [upload=438249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:48:29.479932318 (499h19m29.351015353s) [seg_pos={0 0}, upload=438249]: Begin upload segment... Dec 26 10:48:29.480028431 (499h19m29.351111466s) [seg_pos={0 0}, upload=438249]: Done begin upload segment. Dec 26 10:48:29.480059641 (499h19m29.351142676s) [noise=true , address=145.79.227.42:4121 , piece_id=P52JO367NFGXBMVMD3QD5HMXDWTLS2BSIIWBVYMWW2SR4V7S6CDA, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.480090810 (499h19m29.351173845s) [noise=true , address=145.79.227.2:4115 , piece_id=I6AW3APASHEDNJZOY32XJ5XXB23XX4AMW434TVDYZMFZRFE4QUAA, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.480887898 (499h19m29.351970943s) [noise=true , address=145.79.227.7:4293 , piece_id=HLPCOI4QTV4G4KYL5CSFOZ7BAPOFM65RO4XEKBLCSRZZYTXYS3KQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.481131622 (499h19m29.352214657s) [noise=true , address=45.91.116.8:4046 , piece_id=VFTSK6DKBSVLHEO6S66TOCXJCPOMPURBSSCMA7BZ34TWS6OBAEPA, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.481363905 (499h19m29.35244694s) [noise=true , address=145.79.227.36:4517 , piece_id=NCLEUPNI52542EMTJLLVQC3XQA266O6ZO67ORA45XMCPLU3XN4JA, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.481638707 (499h19m29.352721752s) [noise=true , address=145.79.227.38:4978 , piece_id=6XHQU5TQGAL4VDGXIAPF7PTFFITWF6RDQOUU2CHHJDBWQ5BDB2FQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.481866591 (499h19m29.352949637s) [noise=true , address=145.79.227.37:4566 , piece_id=7F2UNYEIO6CDTI6GXTTTYSVERA555GXIP77KQ7L4XOYIFWT76YAQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.482385139 (499h19m29.353468184s) [noise=true , address=145.79.227.41:4086 , piece_id=L7AVKWLVT3XEANBHQ22GXBVSGNIVYHBT5C2QN7PRMKBINNXHVS6Q, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.482408293 (499h19m29.353491328s) [noise=true , address=77.90.31.11:4555 , piece_id=D3VF2RDY7YAXSKXKR36BUN2PN2O74DWUGXF3GTXL24GEQITFGDOA, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.483095000 (499h19m29.354178035s) [noise=true , address=45.91.116.11:4119 , piece_id=UCWYY5AM2UNNLOLA5BJCW25ZEHZWC5PANSE7UUTB3KHP5FBIYTHQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.483611003 (499h19m29.354694048s) [noise=true , address=145.79.227.11:4731 , piece_id=KVUSMU3Q5S7H54HT4YEJLSXN7RHVO3GCAZE4GNHILIHT64T5DHJQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.484064737 (499h19m29.355147772s) [noise=true , address=45.91.116.13:4155 , piece_id=3ZZL5QWG554YFPY5EVZXDTRJIIDYXRQN5KOMBNYJ5YWT4KZKOJJQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.484487291 (499h19m29.355570326s) [noise=true , address=77.90.31.16:4736 , piece_id=IWLMVZQMEHJKEVDHXCBB6YRI4AGQWKLQPRQ6KSZ5YVFTDD5GIDZQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.484983267 (499h19m29.356066302s) [noise=true , address=45.91.116.4:4939 , piece_id=ZDAJWSYTY7RBS52MNIPHKGU2OVIVORGNOVIN6DZQVHEROEDYV63A, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.485453211 (499h19m29.356536256s) [noise=true , address=77.90.31.17:4773 , piece_id=CW74JZIDSOPD4KUQH4Z7HIOLQKEWIOH5HCSRULIHIBJL56ATLVWQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.486083812 (499h19m29.357166857s) [noise=true , address=145.79.227.24:4583 , piece_id=BC3CKSR4GUHTGQLRTYNV6RYIJSHV2MUKLY4M2HNVCHPZMKJ5R7XA, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:31.184839784 (499h19m31.055922829s) [upload=438249]: Next returned nil segment Dec 26 10:48:31.184857097 (499h19m31.055940142s) [upload=438249]: Waiting for error group managing segments... Dec 26 10:48:31.473297013 (499h19m31.344380059s) [noise=true , address=145.79.227.42:4121 , piece_id=P52JO367NFGXBMVMD3QD5HMXDWTLS2BSIIWBVYMWW2SR4V7S6CDA, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.480617443 (499h19m31.351700529s) [noise=true , address=145.79.227.7:4293 , piece_id=HLPCOI4QTV4G4KYL5CSFOZ7BAPOFM65RO4XEKBLCSRZZYTXYS3KQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.484237266 (499h19m31.355320311s) [noise=true , address=145.79.227.24:4583 , piece_id=BC3CKSR4GUHTGQLRTYNV6RYIJSHV2MUKLY4M2HNVCHPZMKJ5R7XA, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.484809656 (499h19m31.355892701s) [noise=true , address=145.79.227.2:4115 , piece_id=I6AW3APASHEDNJZOY32XJ5XXB23XX4AMW434TVDYZMFZRFE4QUAA, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.489948461 (499h19m31.361031496s) [noise=true , address=145.79.227.41:4086 , piece_id=L7AVKWLVT3XEANBHQ22GXBVSGNIVYHBT5C2QN7PRMKBINNXHVS6Q, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.491852636 (499h19m31.362935672s) [noise=true , address=145.79.227.37:4566 , piece_id=7F2UNYEIO6CDTI6GXTTTYSVERA555GXIP77KQ7L4XOYIFWT76YAQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.493344618 (499h19m31.364427653s) [noise=true , address=145.79.227.11:4731 , piece_id=KVUSMU3Q5S7H54HT4YEJLSXN7RHVO3GCAZE4GNHILIHT64T5DHJQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.504960396 (499h19m31.376043441s) [noise=true , address=145.79.227.36:4517 , piece_id=NCLEUPNI52542EMTJLLVQC3XQA266O6ZO67ORA45XMCPLU3XN4JA, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.513036585 (499h19m31.38411962s) [noise=true , address=145.79.227.38:4978 , piece_id=6XHQU5TQGAL4VDGXIAPF7PTFFITWF6RDQOUU2CHHJDBWQ5BDB2FQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.712335553 (499h19m31.583418588s) [noise=true , address=45.91.116.4:4939 , piece_id=ZDAJWSYTY7RBS52MNIPHKGU2OVIVORGNOVIN6DZQVHEROEDYV63A, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.789317539 (499h19m31.660400584s) [noise=true , address=77.90.31.16:4736 , piece_id=IWLMVZQMEHJKEVDHXCBB6YRI4AGQWKLQPRQ6KSZ5YVFTDD5GIDZQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.821355825 (499h19m31.69243886s) [noise=true , address=45.91.116.13:4155 , piece_id=3ZZL5QWG554YFPY5EVZXDTRJIIDYXRQN5KOMBNYJ5YWT4KZKOJJQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.848404300 (499h19m31.719487345s) [noise=true , address=45.91.116.8:4046 , piece_id=VFTSK6DKBSVLHEO6S66TOCXJCPOMPURBSSCMA7BZ34TWS6OBAEPA, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.848423306 (499h19m31.719506342s) [seg_pos={0 0}, upload=438249]: Segment reached optimal threshold of 13 pieces. Dec 26 10:48:31.848591367 (499h19m31.719674402s) [noise=true , address=77.90.31.17:4773 , piece_id=CW74JZIDSOPD4KUQH4Z7HIOLQKEWIOH5HCSRULIHIBJL56ATLVWQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 10:48:31.848597338 (499h19m31.719680373s) [noise=true , address=77.90.31.17:4773 , piece_id=CW74JZIDSOPD4KUQH4Z7HIOLQKEWIOH5HCSRULIHIBJL56ATLVWQ, seg_pos={0 0}, upload=438249]: Long tail cancellation due to optimal threshold Dec 26 10:48:31.848796858 (499h19m31.719879903s) [noise=true , address=45.91.116.11:4119 , piece_id=UCWYY5AM2UNNLOLA5BJCW25ZEHZWC5PANSE7UUTB3KHP5FBIYTHQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 10:48:31.848812167 (499h19m31.719895202s) [noise=true , address=45.91.116.11:4119 , piece_id=UCWYY5AM2UNNLOLA5BJCW25ZEHZWC5PANSE7UUTB3KHP5FBIYTHQ, seg_pos={0 0}, upload=438249]: Long tail cancellation due to optimal threshold Dec 26 10:48:31.848876439 (499h19m31.719959474s) [noise=true , address=77.90.31.11:4555 , piece_id=D3VF2RDY7YAXSKXKR36BUN2PN2O74DWUGXF3GTXL24GEQITFGDOA, seg_pos={0 0}, upload=438249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 26 10:48:31.848883583 (499h19m31.719966618s) [noise=true , address=77.90.31.11:4555 , piece_id=D3VF2RDY7YAXSKXKR36BUN2PN2O74DWUGXF3GTXL24GEQITFGDOA, seg_pos={0 0}, upload=438249]: Long tail cancellation due to optimal threshold Dec 26 10:48:31.848910614 (499h19m31.719993649s) [seg_pos={0 0}, upload=438249]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:48:31.849075629 (499h19m31.720158664s) [upload=438249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:48:31.849081510 (499h19m31.720164555s) [upload=438249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:48:32.140485121 (499h19m32.011568156s) [upload=438249]: Done uploading segments... Dec 26 10:48:32.140503606 (499h19m32.011586641s) [upload=438249]: Upload finished. err: <nil> Duration: 6.046365037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..XX...X.X.XX....XX.......X.X.....X..X....X..X..X.......XX] duration: 30.00249263s trace: https://jaeger.dev.storj.io/trace/483849bb45a54e20 FAIL (2025-12-26 10:48:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 10:48:29.228035124 (499h19m29.099118169s) [upload=438249]: Starting upload Dec 26 10:48:29.228196011 (499h19m29.099279066s) [upload=438249]: Done starting upload Dec 26 10:48:29.228254632 (499h19m29.099337667s) [upload=438249]: Uploading segments... Dec 26 10:48:29.228570444 (499h19m29.099653479s) [upload=438249]: Got next segment. Inline: false Dec 26 10:48:29.228613005 (499h19m29.09969605s) [upload=438249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:48:29.228618926 (499h19m29.099701971s) [upload=438249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:48:29.479932318 (499h19m29.351015353s) [seg_pos={0 0}, upload=438249]: Begin upload segment... Dec 26 10:48:29.480028431 (499h19m29.351111466s) [seg_pos={0 0}, upload=438249]: Done begin upload segment. Dec 26 10:48:29.480059641 (499h19m29.351142676s) [noise=true , address=145.79.227.42:4121 , piece_id=P52JO367NFGXBMVMD3QD5HMXDWTLS2BSIIWBVYMWW2SR4V7S6CDA, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.480090810 (499h19m29.351173845s) [noise=true , address=145.79.227.2:4115 , piece_id=I6AW3APASHEDNJZOY32XJ5XXB23XX4AMW434TVDYZMFZRFE4QUAA, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.480887898 (499h19m29.351970943s) [noise=true , address=145.79.227.7:4293 , piece_id=HLPCOI4QTV4G4KYL5CSFOZ7BAPOFM65RO4XEKBLCSRZZYTXYS3KQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.481131622 (499h19m29.352214657s) [noise=true , address=45.91.116.8:4046 , piece_id=VFTSK6DKBSVLHEO6S66TOCXJCPOMPURBSSCMA7BZ34TWS6OBAEPA, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.481363905 (499h19m29.35244694s) [noise=true , address=145.79.227.36:4517 , piece_id=NCLEUPNI52542EMTJLLVQC3XQA266O6ZO67ORA45XMCPLU3XN4JA, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.481638707 (499h19m29.352721752s) [noise=true , address=145.79.227.38:4978 , piece_id=6XHQU5TQGAL4VDGXIAPF7PTFFITWF6RDQOUU2CHHJDBWQ5BDB2FQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.481866591 (499h19m29.352949637s) [noise=true , address=145.79.227.37:4566 , piece_id=7F2UNYEIO6CDTI6GXTTTYSVERA555GXIP77KQ7L4XOYIFWT76YAQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.482385139 (499h19m29.353468184s) [noise=true , address=145.79.227.41:4086 , piece_id=L7AVKWLVT3XEANBHQ22GXBVSGNIVYHBT5C2QN7PRMKBINNXHVS6Q, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.482408293 (499h19m29.353491328s) [noise=true , address=77.90.31.11:4555 , piece_id=D3VF2RDY7YAXSKXKR36BUN2PN2O74DWUGXF3GTXL24GEQITFGDOA, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.483095000 (499h19m29.354178035s) [noise=true , address=45.91.116.11:4119 , piece_id=UCWYY5AM2UNNLOLA5BJCW25ZEHZWC5PANSE7UUTB3KHP5FBIYTHQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.483611003 (499h19m29.354694048s) [noise=true , address=145.79.227.11:4731 , piece_id=KVUSMU3Q5S7H54HT4YEJLSXN7RHVO3GCAZE4GNHILIHT64T5DHJQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.484064737 (499h19m29.355147772s) [noise=true , address=45.91.116.13:4155 , piece_id=3ZZL5QWG554YFPY5EVZXDTRJIIDYXRQN5KOMBNYJ5YWT4KZKOJJQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.484487291 (499h19m29.355570326s) [noise=true , address=77.90.31.16:4736 , piece_id=IWLMVZQMEHJKEVDHXCBB6YRI4AGQWKLQPRQ6KSZ5YVFTDD5GIDZQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.484983267 (499h19m29.356066302s) [noise=true , address=45.91.116.4:4939 , piece_id=ZDAJWSYTY7RBS52MNIPHKGU2OVIVORGNOVIN6DZQVHEROEDYV63A, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.485453211 (499h19m29.356536256s) [noise=true , address=77.90.31.17:4773 , piece_id=CW74JZIDSOPD4KUQH4Z7HIOLQKEWIOH5HCSRULIHIBJL56ATLVWQ, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:29.486083812 (499h19m29.357166857s) [noise=true , address=145.79.227.24:4583 , piece_id=BC3CKSR4GUHTGQLRTYNV6RYIJSHV2MUKLY4M2HNVCHPZMKJ5R7XA, seg_pos={0 0}, upload=438249]: Uploading piece... Dec 26 10:48:31.184839784 (499h19m31.055922829s) [upload=438249]: Next returned nil segment Dec 26 10:48:31.184857097 (499h19m31.055940142s) [upload=438249]: Waiting for error group managing segments... Dec 26 10:48:31.473297013 (499h19m31.344380059s) [noise=true , address=145.79.227.42:4121 , piece_id=P52JO367NFGXBMVMD3QD5HMXDWTLS2BSIIWBVYMWW2SR4V7S6CDA, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.480617443 (499h19m31.351700529s) [noise=true , address=145.79.227.7:4293 , piece_id=HLPCOI4QTV4G4KYL5CSFOZ7BAPOFM65RO4XEKBLCSRZZYTXYS3KQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.484237266 (499h19m31.355320311s) [noise=true , address=145.79.227.24:4583 , piece_id=BC3CKSR4GUHTGQLRTYNV6RYIJSHV2MUKLY4M2HNVCHPZMKJ5R7XA, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.484809656 (499h19m31.355892701s) [noise=true , address=145.79.227.2:4115 , piece_id=I6AW3APASHEDNJZOY32XJ5XXB23XX4AMW434TVDYZMFZRFE4QUAA, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.489948461 (499h19m31.361031496s) [noise=true , address=145.79.227.41:4086 , piece_id=L7AVKWLVT3XEANBHQ22GXBVSGNIVYHBT5C2QN7PRMKBINNXHVS6Q, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.491852636 (499h19m31.362935672s) [noise=true , address=145.79.227.37:4566 , piece_id=7F2UNYEIO6CDTI6GXTTTYSVERA555GXIP77KQ7L4XOYIFWT76YAQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.493344618 (499h19m31.364427653s) [noise=true , address=145.79.227.11:4731 , piece_id=KVUSMU3Q5S7H54HT4YEJLSXN7RHVO3GCAZE4GNHILIHT64T5DHJQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.504960396 (499h19m31.376043441s) [noise=true , address=145.79.227.36:4517 , piece_id=NCLEUPNI52542EMTJLLVQC3XQA266O6ZO67ORA45XMCPLU3XN4JA, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.513036585 (499h19m31.38411962s) [noise=true , address=145.79.227.38:4978 , piece_id=6XHQU5TQGAL4VDGXIAPF7PTFFITWF6RDQOUU2CHHJDBWQ5BDB2FQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.712335553 (499h19m31.583418588s) [noise=true , address=45.91.116.4:4939 , piece_id=ZDAJWSYTY7RBS52MNIPHKGU2OVIVORGNOVIN6DZQVHEROEDYV63A, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.789317539 (499h19m31.660400584s) [noise=true , address=77.90.31.16:4736 , piece_id=IWLMVZQMEHJKEVDHXCBB6YRI4AGQWKLQPRQ6KSZ5YVFTDD5GIDZQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.821355825 (499h19m31.69243886s) [noise=true , address=45.91.116.13:4155 , piece_id=3ZZL5QWG554YFPY5EVZXDTRJIIDYXRQN5KOMBNYJ5YWT4KZKOJJQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.848404300 (499h19m31.719487345s) [noise=true , address=45.91.116.8:4046 , piece_id=VFTSK6DKBSVLHEO6S66TOCXJCPOMPURBSSCMA7BZ34TWS6OBAEPA, seg_pos={0 0}, upload=438249]: Done uploading piece. err: <nil> Dec 26 10:48:31.848423306 (499h19m31.719506342s) [seg_pos={0 0}, upload=438249]: Segment reached optimal threshold of 13 pieces. Dec 26 10:48:31.848591367 (499h19m31.719674402s) [noise=true , address=77.90.31.17:4773 , piece_id=CW74JZIDSOPD4KUQH4Z7HIOLQKEWIOH5HCSRULIHIBJL56ATLVWQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 10:48:31.848597338 (499h19m31.719680373s) [noise=true , address=77.90.31.17:4773 , piece_id=CW74JZIDSOPD4KUQH4Z7HIOLQKEWIOH5HCSRULIHIBJL56ATLVWQ, seg_pos={0 0}, upload=438249]: Long tail cancellation due to optimal threshold Dec 26 10:48:31.848796858 (499h19m31.719879903s) [noise=true , address=45.91.116.11:4119 , piece_id=UCWYY5AM2UNNLOLA5BJCW25ZEHZWC5PANSE7UUTB3KHP5FBIYTHQ, seg_pos={0 0}, upload=438249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 10:48:31.848812167 (499h19m31.719895202s) [noise=true , address=45.91.116.11:4119 , piece_id=UCWYY5AM2UNNLOLA5BJCW25ZEHZWC5PANSE7UUTB3KHP5FBIYTHQ, seg_pos={0 0}, upload=438249]: Long tail cancellation due to optimal threshold Dec 26 10:48:31.848876439 (499h19m31.719959474s) [noise=true , address=77.90.31.11:4555 , piece_id=D3VF2RDY7YAXSKXKR36BUN2PN2O74DWUGXF3GTXL24GEQITFGDOA, seg_pos={0 0}, upload=438249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 26 10:48:31.848883583 (499h19m31.719966618s) [noise=true , address=77.90.31.11:4555 , piece_id=D3VF2RDY7YAXSKXKR36BUN2PN2O74DWUGXF3GTXL24GEQITFGDOA, seg_pos={0 0}, upload=438249]: Long tail cancellation due to optimal threshold Dec 26 10:48:31.848910614 (499h19m31.719993649s) [seg_pos={0 0}, upload=438249]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:48:31.849075629 (499h19m31.720158664s) [upload=438249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:48:31.849081510 (499h19m31.720164555s) [upload=438249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:48:32.140485121 (499h19m32.011568156s) [upload=438249]: Done uploading segments... Dec 26 10:48:32.140503606 (499h19m32.011586641s) [upload=438249]: Upload finished. err: <nil> Duration: 30.002502629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............X.......XXX......X............X.........X....] duration: 34.637210811s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 10:48:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 34.637230408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X..XX...X.X.XX....XX.......X.X.....X..X....X..X..X.......X] duration: 30.003462197s trace: https://jaeger.dev.storj.io/trace/7376e80c09acb2a0 FAIL (2025-12-26 10:48:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: NL4CI2QIE2W267HH6K4GKBGM4WDQFDMT7T2G63AD6PWBOK22T6KQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: YLFQCZJAWYQQXYZM7X7KJZVZEJS4T6EZ3L6O6XOYNF6SOIFEZ2VA) context deadline exceeded ======LOGS====== Dec 26 10:47:59.009481397 (499h18m58.880564462s) [upload=438239]: Starting upload Dec 26 10:47:59.009556229 (499h18m58.880639265s) [upload=438239]: Done starting upload Dec 26 10:47:59.009650870 (499h18m58.880733905s) [upload=438239]: Uploading segments... Dec 26 10:47:59.009829260 (499h18m58.880912295s) [upload=438239]: Got next segment. Inline: false Dec 26 10:47:59.009929270 (499h18m58.881012305s) [upload=438239]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:47:59.009934480 (499h18m58.881017515s) [upload=438239]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:47:59.253347238 (499h18m59.124430283s) [seg_pos={0 0}, upload=438239]: Begin upload segment... Dec 26 10:47:59.253399588 (499h18m59.124482634s) [seg_pos={0 0}, upload=438239]: Done begin upload segment. Dec 26 10:47:59.253428213 (499h18m59.124511248s) [noise=true , address=145.79.227.6:4257 , piece_id=UJNJIPT7MOFFY7GKIMTQH3D7ETF3CCSNB7NZDCDSF7JEGWBLCYUA, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.253475042 (499h18m59.124558087s) [noise=true , address=145.79.227.52:4447 , piece_id=IZ4DF37VHWKQX3ET5A7ZVNM7SQF4D2JOYIYMEHJ2S5RBKF6X7WTQ, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.254170707 (499h18m59.125253742s) [noise=true , address=45.91.116.9:4263 , piece_id=NL4CI2QIE2W267HH6K4GKBGM4WDQFDMT7T2G63AD6PWBOK22T6KQ, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.254515854 (499h18m59.125598889s) [noise=true , address=77.90.31.18:4808 , piece_id=KHNHLKXJFWCGNSKXJQEPYEIVQO2ZJ63SFVYZ5S3ASMHWB2IDLA4A, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.254720704 (499h18m59.125803749s) [noise=true , address=145.79.227.46:4602 , piece_id=SO3HCD4LKZQ5NGHKCCRMQ2ZD53GWTZH7VRXZYZTRD2SLA3GTO23Q, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.254840802 (499h18m59.125923837s) [noise=true , address=145.79.227.13:4802 , piece_id=NG2ECR6ZM4MEIAYY65C7ZO3JSNABE4U2KHO7SPGRWIO6Q66U4OCQ, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.255143589 (499h18m59.126226624s) [noise=true , address=45.91.116.13:4155 , piece_id=YLFQCZJAWYQQXYZM7X7KJZVZEJS4T6EZ3L6O6XOYNF6SOIFEZ2VA, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.255291090 (499h18m59.126374135s) [noise=true , address=145.79.227.34:4906 , piece_id=B6XKETFWR47KIW3VKUIVA7TMLMQZKAMO37RYBMBCMC4YJPWL77GQ, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.255480551 (499h18m59.126563586s) [noise=true , address=145.79.227.2:4114 , piece_id=P7BCLURZKGCFCPGBAIX2PWKVOMVYKHL4TJBVC435TJZYEHRXXK5Q, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.255793517 (499h18m59.126876552s) [noise=true , address=145.79.227.43:4157 , piece_id=RAFDAQR26SXJAAEB2KX76KSBNR6OPZQM5XD3MOCT53UEL2ZLWE3Q, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.257768948 (499h18m59.128851993s) [noise=true , address=45.91.116.10:4082 , piece_id=SMR7VB2VBD23JV5MX5GH4ZWEEHWJKQSS5QJQETIUQ3CKP5YLP6ZA, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.258152037 (499h18m59.129235073s) [noise=true , address=145.79.227.19:4078 , piece_id=54DSQDDNM6YERRGKE3X26DMCS26SKO7LR25VXD2Y7CTQZNQKZXUQ, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.258638714 (499h18m59.129721759s) [noise=true , address=45.91.116.12:4225 , piece_id=HQBDDEWMCI46ZSK4TOONV3HR5S4XOWZGO74SXAY2XHGB2TOZD4OA, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.259038526 (499h18m59.130121571s) [noise=true , address=77.90.31.16:4735 , piece_id=TQTVMZGKWMKZKASBKWDZUN6F4FP7ZZFYJDFXECT5SJEEEWXSD6AA, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.259896790 (499h18m59.130979835s) [noise=true , address=77.90.31.11:4556 , piece_id=CDWVAW2J5ZBFPDBWQJ3ZQ7CSGCDLNWRPB2YCN6OGCJKDJR7FCUTA, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:47:59.261422946 (499h18m59.132505991s) [noise=true , address=45.91.116.14:4190 , piece_id=SH5JVULPKQEIOPKHDCJGSFIAN436ROPDSFEHK7BO6LA6P3NL5VVQ, seg_pos={0 0}, upload=438239]: Uploading piece... Dec 26 10:48:00.975413266 (499h19m0.846496301s) [upload=438239]: Next returned nil segment Dec 26 10:48:00.975425941 (499h19m0.846508976s) [upload=438239]: Waiting for error group managing segments... Dec 26 10:48:01.257117498 (499h19m1.128200543s) [noise=true , address=145.79.227.13:4802 , piece_id=NG2ECR6ZM4MEIAYY65C7ZO3JSNABE4U2KHO7SPGRWIO6Q66U4OCQ, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.260356536 (499h19m1.131439571s) [noise=true , address=145.79.227.6:4257 , piece_id=UJNJIPT7MOFFY7GKIMTQH3D7ETF3CCSNB7NZDCDSF7JEGWBLCYUA, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.262103492 (499h19m1.133186527s) [noise=true , address=145.79.227.46:4602 , piece_id=SO3HCD4LKZQ5NGHKCCRMQ2ZD53GWTZH7VRXZYZTRD2SLA3GTO23Q, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.268084150 (499h19m1.139167195s) [noise=true , address=145.79.227.52:4447 , piece_id=IZ4DF37VHWKQX3ET5A7ZVNM7SQF4D2JOYIYMEHJ2S5RBKF6X7WTQ, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.281894888 (499h19m1.152977933s) [noise=true , address=145.79.227.2:4114 , piece_id=P7BCLURZKGCFCPGBAIX2PWKVOMVYKHL4TJBVC435TJZYEHRXXK5Q, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.284832372 (499h19m1.155915417s) [noise=true , address=145.79.227.19:4078 , piece_id=54DSQDDNM6YERRGKE3X26DMCS26SKO7LR25VXD2Y7CTQZNQKZXUQ, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.296555023 (499h19m1.167638068s) [noise=true , address=145.79.227.43:4157 , piece_id=RAFDAQR26SXJAAEB2KX76KSBNR6OPZQM5XD3MOCT53UEL2ZLWE3Q, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.355941826 (499h19m1.227024871s) [noise=true , address=145.79.227.34:4906 , piece_id=B6XKETFWR47KIW3VKUIVA7TMLMQZKAMO37RYBMBCMC4YJPWL77GQ, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.503386395 (499h19m1.37446944s) [noise=true , address=45.91.116.9:4263 , piece_id=NL4CI2QIE2W267HH6K4GKBGM4WDQFDMT7T2G63AD6PWBOK22T6KQ, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.515174301 (499h19m1.386257356s) [noise=true , address=45.91.116.13:4155 , piece_id=YLFQCZJAWYQQXYZM7X7KJZVZEJS4T6EZ3L6O6XOYNF6SOIFEZ2VA, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.583366109 (499h19m1.454449154s) [noise=true , address=45.91.116.10:4082 , piece_id=SMR7VB2VBD23JV5MX5GH4ZWEEHWJKQSS5QJQETIUQ3CKP5YLP6ZA, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.599779353 (499h19m1.470862398s) [noise=true , address=77.90.31.11:4556 , piece_id=CDWVAW2J5ZBFPDBWQJ3ZQ7CSGCDLNWRPB2YCN6OGCJKDJR7FCUTA, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.679821016 (499h19m1.550904061s) [noise=true , address=77.90.31.16:4735 , piece_id=TQTVMZGKWMKZKASBKWDZUN6F4FP7ZZFYJDFXECT5SJEEEWXSD6AA, seg_pos={0 0}, upload=438239]: Done uploading piece. err: <nil> Dec 26 10:48:01.679843638 (499h19m1.550926683s) [seg_pos={0 0}, upload=438239]: Segment reached optimal threshold of 13 pieces. Dec 26 10:48:01.680042267 (499h19m1.551125312s) [noise=true , address=45.91.116.14:4190 , piece_id=SH5JVULPKQEIOPKHDCJGSFIAN436ROPDSFEHK7BO6LA6P3NL5VVQ, seg_pos={0 0}, upload=438239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 10:48:01.680048969 (499h19m1.551132004s) [noise=true , address=45.91.116.14:4190 , piece_id=SH5JVULPKQEIOPKHDCJGSFIAN436ROPDSFEHK7BO6LA6P3NL5VVQ, seg_pos={0 0}, upload=438239]: Long tail cancellation due to optimal threshold Dec 26 10:48:01.680197963 (499h19m1.551281008s) [noise=true , address=77.90.31.18:4808 , piece_id=KHNHLKXJFWCGNSKXJQEPYEIVQO2ZJ63SFVYZ5S3ASMHWB2IDLA4A, seg_pos={0 0}, upload=438239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:49214->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:49214->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49214->77.90.31.18:4808: use of closed network connection Dec 26 10:48:01.680209405 (499h19m1.55129245s) [noise=true , address=77.90.31.18:4808 , piece_id=KHNHLKXJFWCGNSKXJQEPYEIVQO2ZJ63SFVYZ5S3ASMHWB2IDLA4A, seg_pos={0 0}, upload=438239]: Long tail cancellation due to optimal threshold Dec 26 10:48:01.680249451 (499h19m1.551332496s) [noise=true , address=45.91.116.12:4225 , piece_id=HQBDDEWMCI46ZSK4TOONV3HR5S4XOWZGO74SXAY2XHGB2TOZD4OA, seg_pos={0 0}, upload=438239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 10:48:01.680257676 (499h19m1.551340712s) [noise=true , address=45.91.116.12:4225 , piece_id=HQBDDEWMCI46ZSK4TOONV3HR5S4XOWZGO74SXAY2XHGB2TOZD4OA, seg_pos={0 0}, upload=438239]: Long tail cancellation due to optimal threshold Dec 26 10:48:01.680279028 (499h19m1.551362063s) [seg_pos={0 0}, upload=438239]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:48:01.680362606 (499h19m1.551445651s) [upload=438239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:48:01.680369760 (499h19m1.551452805s) [upload=438239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:48:01.966139463 (499h19m1.837222508s) [upload=438239]: Done uploading segments... Dec 26 10:48:01.966160563 (499h19m1.837243608s) [upload=438239]: Upload finished. err: <nil> Duration: 30.003483257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [................X.......XXX......X............X.........X...] duration: 41.009910271s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 10:48:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 41.009924859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................X.......XXX......X............X.........X..] duration: 36.006041078s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 10:47:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 36.006060975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X..XX...X.X.XX....XX.......X.X.....X..X....X..X..X.......] duration: 14.696904691s trace: https://jaeger.dev.storj.io/trace/4338b564d5adc13a FAIL (2025-12-26 10:47:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ) context deadline exceeded ======LOGS====== Dec 26 10:40:43.709056067 (499h11m43.580139112s) [upload=438134]: Starting upload Dec 26 10:40:43.709110280 (499h11m43.580193325s) [upload=438134]: Done starting upload Dec 26 10:40:43.709167779 (499h11m43.580250815s) [upload=438134]: Uploading segments... Dec 26 10:40:43.709289561 (499h11m43.580372606s) [upload=438134]: Got next segment. Inline: false Dec 26 10:40:43.709316532 (499h11m43.580399568s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:40:43.709321101 (499h11m43.580404146s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:40:43.955318361 (499h11m43.826401406s) [seg_pos={0 0}, upload=438134]: Begin upload segment... Dec 26 10:40:43.955375580 (499h11m43.826458625s) [seg_pos={0 0}, upload=438134]: Done begin upload segment. Dec 26 10:40:43.955413943 (499h11m43.826496988s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.955829925 (499h11m43.82691299s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956269121 (499h11m43.827352156s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956570074 (499h11m43.827653109s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956603287 (499h11m43.827686342s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957175627 (499h11m43.828258692s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957403531 (499h11m43.828486576s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957731555 (499h11m43.8288146s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957757996 (499h11m43.828841041s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957852045 (499h11m43.82893509s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958244603 (499h11m43.829327648s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958689861 (499h11m43.829772906s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958798417 (499h11m43.829881452s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.959579504 (499h11m43.830662549s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960180179 (499h11m43.831263224s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960745525 (499h11m43.83182857s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:45.074144120 (499h11m44.945227155s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 10:40:45.670947462 (499h11m45.542030507s) [upload=438134]: Next returned nil segment Dec 26 10:40:45.670960177 (499h11m45.542043212s) [upload=438134]: Waiting for error group managing segments... Dec 26 10:40:45.954644609 (499h11m45.825727654s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.959526224 (499h11m45.830609259s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.964256672 (499h11m45.835339707s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.971375917 (499h11m45.842458952s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.972293225 (499h11m45.84337627s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.983096866 (499h11m45.854179912s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.985743005 (499h11m45.85682605s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986187622 (499h11m45.857270677s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986484257 (499h11m45.857567302s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.284280851 (499h11m46.155363886s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.326738467 (499h11m46.197821512s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.482782160 (499h11m46.353865205s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153035854 (499h11m47.024118889s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153057365 (499h11m47.0241404s) [seg_pos={0 0}, upload=438134]: Segment reached optimal threshold of 13 pieces. Dec 26 10:40:47.153218621 (499h11m47.024301657s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 10:40:47.153225585 (499h11m47.02430862s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153561755 (499h11m47.0246448s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection Dec 26 10:40:47.153583056 (499h11m47.024666101s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153617862 (499h11m47.024700907s) [seg_pos={0 0}, upload=438134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:40:47.153841508 (499h11m47.024924553s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:40:47.153849653 (499h11m47.024932688s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:40:47.437182455 (499h11m47.3082655s) [upload=438134]: Done uploading segments... Dec 26 10:40:47.437201211 (499h11m47.308284246s) [upload=438134]: Upload finished. err: <nil> Duration: 14.696932715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................X.......XXX......X............X.........X.] duration: 51.137519448s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 10:46:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 51.137537964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X.X..XX...X.X.XX....XX.......X.X.....X..X....X..X..X......] duration: 5.709525818s trace: https://jaeger.dev.storj.io/trace/4338b564d5adc13a FAIL (2025-12-26 10:46:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ) context deadline exceeded ======LOGS====== Dec 26 10:40:43.709056067 (499h11m43.580139112s) [upload=438134]: Starting upload Dec 26 10:40:43.709110280 (499h11m43.580193325s) [upload=438134]: Done starting upload Dec 26 10:40:43.709167779 (499h11m43.580250815s) [upload=438134]: Uploading segments... Dec 26 10:40:43.709289561 (499h11m43.580372606s) [upload=438134]: Got next segment. Inline: false Dec 26 10:40:43.709316532 (499h11m43.580399568s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:40:43.709321101 (499h11m43.580404146s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:40:43.955318361 (499h11m43.826401406s) [seg_pos={0 0}, upload=438134]: Begin upload segment... Dec 26 10:40:43.955375580 (499h11m43.826458625s) [seg_pos={0 0}, upload=438134]: Done begin upload segment. Dec 26 10:40:43.955413943 (499h11m43.826496988s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.955829925 (499h11m43.82691299s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956269121 (499h11m43.827352156s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956570074 (499h11m43.827653109s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956603287 (499h11m43.827686342s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957175627 (499h11m43.828258692s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957403531 (499h11m43.828486576s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957731555 (499h11m43.8288146s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957757996 (499h11m43.828841041s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957852045 (499h11m43.82893509s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958244603 (499h11m43.829327648s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958689861 (499h11m43.829772906s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958798417 (499h11m43.829881452s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.959579504 (499h11m43.830662549s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960180179 (499h11m43.831263224s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960745525 (499h11m43.83182857s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:45.074144120 (499h11m44.945227155s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 10:40:45.670947462 (499h11m45.542030507s) [upload=438134]: Next returned nil segment Dec 26 10:40:45.670960177 (499h11m45.542043212s) [upload=438134]: Waiting for error group managing segments... Dec 26 10:40:45.954644609 (499h11m45.825727654s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.959526224 (499h11m45.830609259s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.964256672 (499h11m45.835339707s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.971375917 (499h11m45.842458952s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.972293225 (499h11m45.84337627s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.983096866 (499h11m45.854179912s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.985743005 (499h11m45.85682605s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986187622 (499h11m45.857270677s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986484257 (499h11m45.857567302s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.284280851 (499h11m46.155363886s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.326738467 (499h11m46.197821512s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.482782160 (499h11m46.353865205s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153035854 (499h11m47.024118889s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153057365 (499h11m47.0241404s) [seg_pos={0 0}, upload=438134]: Segment reached optimal threshold of 13 pieces. Dec 26 10:40:47.153218621 (499h11m47.024301657s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 10:40:47.153225585 (499h11m47.02430862s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153561755 (499h11m47.0246448s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection Dec 26 10:40:47.153583056 (499h11m47.024666101s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153617862 (499h11m47.024700907s) [seg_pos={0 0}, upload=438134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:40:47.153841508 (499h11m47.024924553s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:40:47.153849653 (499h11m47.024932688s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:40:47.437182455 (499h11m47.3082655s) [upload=438134]: Done uploading segments... Dec 26 10:40:47.437201211 (499h11m47.308284246s) [upload=438134]: Upload finished. err: <nil> Duration: 5.709555426s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..................X.......XXX......X............X.........X] duration: 1m0.002709181s trace: https://jaeger.dev.storj.io/trace/785a90e8be84a595 FAIL (2025-12-26 10:45:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KXVRQQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834123964 (499h15m56.705207009s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834585043 (499h15m56.705668078s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.834961800 (499h15m56.706044845s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835331334 (499h15m56.706414379s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.835738399 (499h15m56.706821434s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836118653 (499h15m56.707201698s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836501923 (499h15m56.707584968s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.836904940 (499h15m56.707987985s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837388641 (499h15m56.708471676s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.837910455 (499h15m56.7089935s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838146514 (499h15m56.709229559s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838469329 (499h15m56.709552364s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.838690831 (499h15m56.709773866s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839143724 (499h15m56.710226759s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:56.839678982 (499h15m56.710762027s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Uploading piece... Dec 26 10:44:58.136536405 (499h15m58.00761944s) [upload=438194]: Next returned nil segment Dec 26 10:44:58.136548628 (499h15m58.007631673s) [upload=438194]: Waiting for error group managing segments... Dec 26 10:44:58.469618858 (499h15m58.340701893s) [noise=true , address=64.49.13.26:4945 , piece_id=KXCFTLWR4HSBTP4GJJ4HEPA6BBEMILYTJEMXY4S5WB24HRMQJOKA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.708819055 (499h15m58.5799021s) [noise=true , address=145.79.227.9:4390 , piece_id=JDQNHNGABBGANIHKSKP4VHUWIPQBNNEOPC4ZQCKR3K5TTR4N4SGQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.714466429 (499h15m58.585549474s) [noise=true , address=145.79.227.40:4076 , piece_id=Y6WKUYNPDMBUZSNWEYDL2KKNR6WKHSKGP63S6KAG6F2MWMNTVMBQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.745407696 (499h15m58.616490731s) [noise=true , address=145.79.227.20:4459 , piece_id=2EUW2AH7FBVZNJ27XVLRXKMXA737DWKAYBL7CR7PRXXRTO2M644A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.888701070 (499h15m58.759784105s) [noise=true , address=77.90.31.18:4838 , piece_id=VYBIKY3MLEE2CXGS2MT23HX5HUFK25FKYKI4XLXPDRNWCC55ANXQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.948150873 (499h15m58.819233918s) [noise=true , address=45.91.116.5:4301 , piece_id=E3ZRKZUKM335ARHV3Z5YGLKQYHG6QV4GCCNHAVMN34BM7UQFWIIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:58.971389744 (499h15m58.842472779s) [noise=true , address=77.90.31.19:4868 , piece_id=WH77PZSXC2VUWLQYLBR2NDEVT6KMD3VZK5WAL53AMRDW74P242LQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.021685314 (499h15m58.892768349s) [noise=true , address=45.91.116.4:4963 , piece_id=U4T64KQ67C3BDKG5TJSTQWUS4WWYCSVPUZOWJ6YANR2IQ2JD4EFQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.045418216 (499h15m58.916501251s) [noise=true , address=45.91.116.14:4204 , piece_id=JWL7OOMKTVAP7WEUB5WTL3F6F5IRNID4KIOYJSYZ655PJSNXAQKQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.056260371 (499h15m58.927343416s) [noise=true , address=45.91.116.9:4272 , piece_id=GGFDVKEW5LUCHQQUP6CWJHJQXUETYUG2UISPVP6XQJGSAJW2KP2Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.085713054 (499h15m58.956796089s) [noise=true , address=45.91.116.7:4017 , piece_id=L4GGY6AICMSFMUVYGI6ZMUORYRXNJFPLJY5XJRUAM7NVYQOHPHSQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.640980124 (499h15m59.512063169s) [noise=true , address=145.79.227.30:4778 , piece_id=BMEAUWANP7FP6CFRULMIAZUV4JNQEXOUN2EAXJY5XOWLDZINWOIA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.654515607 (499h15m59.525598652s) [noise=true , address=145.79.227.44:4208 , piece_id=ZLIX7XAUMKRUEWWTQST2KLCOWFJSAFSAQNLUCTTOWVZUBESEI23A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660349657 (499h15m59.531432712s) [noise=true , address=77.90.31.11:4571 , piece_id=OERZACIKIG7HBEV3LL42KUAGKN6AWZEXNQSA52FIYTUKDFWCBW7Q, seg_pos={0 0}, upload=438194]: Done uploading piece. err: <nil> Dec 26 10:44:59.660376087 (499h15m59.531459132s) [seg_pos={0 0}, upload=438194]: Segment reached optimal threshold of 14 pieces. Dec 26 10:44:59.660612988 (499h15m59.531696033s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52964->64.49.13.169:4711: use of closed network connection Dec 26 10:44:59.660620753 (499h15m59.531703788s) [noise=true , address=64.49.13.169:4711 , piece_id=PVBVY4TVJLPKYBP6A7LBNZADLN25XBDWZFEK2QVN45YHJATIR7TA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.660893663 (499h15m59.531976698s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.660901057 (499h15m59.531984102s) [noise=true , address=45.91.116.6:4988 , piece_id=VPEKJF74D7SQWKR47MAIY5UJQPBJIEU5AQDKLFGATYIRHFREAQ5A, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661226155 (499h15m59.5323092s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:35884->77.90.31.14:4678: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:44:59.661249630 (499h15m59.532332685s) [noise=true , address=77.90.31.14:4678 , piece_id=XQHQZJUPB24G3BWGFTFDD42NM6K5HJVHUXBXWH5TW7BU25QGHTMA, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661309705 (499h15m59.53239275s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 26 10:44:59.661318962 (499h15m59.532401997s) [noise=true , address=77.90.31.21:4909 , piece_id=H6KT2AKZISCU4VEA6TXJJ3EFGCRBZPPXBEJJEIHZR3GEJZKXVRQQ, seg_pos={0 0}, upload=438194]: Long tail cancellation due to optimal threshold Dec 26 10:44:59.661345674 (499h15m59.532428719s) [seg_pos={0 0}, upload=438194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:44:59.661485770 (499h15m59.532568815s) [upload=438194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:44:59.661493655 (499h15m59.53257669s) [upload=438194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:44:59.968467153 (499h15m59.839550198s) [upload=438194]: Done uploading segments... Dec 26 10:44:59.968484155 (499h15m59.83956719s) [upload=438194]: Upload finished. err: <nil> Duration: 1m0.002718668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX.X.X..XX...X.X.XX....XX.......X.X.....X..X....X..X..X.....] duration: 19.139209312s trace: https://jaeger.dev.storj.io/trace/4338b564d5adc13a FAIL (2025-12-26 10:45:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ) context deadline exceeded ======LOGS====== Dec 26 10:40:43.709056067 (499h11m43.580139112s) [upload=438134]: Starting upload Dec 26 10:40:43.709110280 (499h11m43.580193325s) [upload=438134]: Done starting upload Dec 26 10:40:43.709167779 (499h11m43.580250815s) [upload=438134]: Uploading segments... Dec 26 10:40:43.709289561 (499h11m43.580372606s) [upload=438134]: Got next segment. Inline: false Dec 26 10:40:43.709316532 (499h11m43.580399568s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:40:43.709321101 (499h11m43.580404146s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:40:43.955318361 (499h11m43.826401406s) [seg_pos={0 0}, upload=438134]: Begin upload segment... Dec 26 10:40:43.955375580 (499h11m43.826458625s) [seg_pos={0 0}, upload=438134]: Done begin upload segment. Dec 26 10:40:43.955413943 (499h11m43.826496988s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.955829925 (499h11m43.82691299s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956269121 (499h11m43.827352156s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956570074 (499h11m43.827653109s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956603287 (499h11m43.827686342s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957175627 (499h11m43.828258692s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957403531 (499h11m43.828486576s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957731555 (499h11m43.8288146s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957757996 (499h11m43.828841041s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957852045 (499h11m43.82893509s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958244603 (499h11m43.829327648s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958689861 (499h11m43.829772906s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958798417 (499h11m43.829881452s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.959579504 (499h11m43.830662549s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960180179 (499h11m43.831263224s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960745525 (499h11m43.83182857s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:45.074144120 (499h11m44.945227155s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 10:40:45.670947462 (499h11m45.542030507s) [upload=438134]: Next returned nil segment Dec 26 10:40:45.670960177 (499h11m45.542043212s) [upload=438134]: Waiting for error group managing segments... Dec 26 10:40:45.954644609 (499h11m45.825727654s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.959526224 (499h11m45.830609259s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.964256672 (499h11m45.835339707s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.971375917 (499h11m45.842458952s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.972293225 (499h11m45.84337627s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.983096866 (499h11m45.854179912s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.985743005 (499h11m45.85682605s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986187622 (499h11m45.857270677s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986484257 (499h11m45.857567302s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.284280851 (499h11m46.155363886s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.326738467 (499h11m46.197821512s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.482782160 (499h11m46.353865205s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153035854 (499h11m47.024118889s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153057365 (499h11m47.0241404s) [seg_pos={0 0}, upload=438134]: Segment reached optimal threshold of 13 pieces. Dec 26 10:40:47.153218621 (499h11m47.024301657s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 10:40:47.153225585 (499h11m47.02430862s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153561755 (499h11m47.0246448s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection Dec 26 10:40:47.153583056 (499h11m47.024666101s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153617862 (499h11m47.024700907s) [seg_pos={0 0}, upload=438134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:40:47.153841508 (499h11m47.024924553s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:40:47.153849653 (499h11m47.024932688s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:40:47.437182455 (499h11m47.3082655s) [upload=438134]: Done uploading segments... Dec 26 10:40:47.437201211 (499h11m47.308284246s) [upload=438134]: Upload finished. err: <nil> Duration: 19.139243488s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X..................X.......XXX......X............X.........] duration: 31.494976092s trace: https://jaeger.dev.storj.io/trace/689c45a79c742610 FAIL (2025-12-26 10:44:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 10:36:27.577470372 (499h7m27.448553417s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.577672456 (499h7m27.448755491s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578119137 (499h7m27.449202172s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578562491 (499h7m27.449645536s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578686798 (499h7m27.449769843s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578965638 (499h7m27.450048684s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579314914 (499h7m27.450397959s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579456253 (499h7m27.450539328s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579866264 (499h7m27.450949299s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580237681 (499h7m27.451320716s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580270013 (499h7m27.451353048s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580573590 (499h7m27.451656635s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580865527 (499h7m27.451948562s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.581836275 (499h7m27.45291931s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.583318146 (499h7m27.454401191s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.585237882 (499h7m27.456320937s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:28.960965352 (499h7m28.832048387s) [upload=438077]: Next returned nil segment Dec 26 10:36:28.960982154 (499h7m28.832065199s) [upload=438077]: Waiting for error group managing segments... Dec 26 10:36:29.281740992 (499h7m29.152824037s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.457737227 (499h7m29.328820262s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.468672399 (499h7m29.339755434s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.474132455 (499h7m29.34521549s) [noise=true , address=145.79.227.49:4354 , piece_id=WXFH2GU2GMTNYG2FC3UEORTY2LCBUOYB3WXORXZ4RTSZZJFGIDCQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.563457636 (499h7m29.434540681s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.600957240 (499h7m29.472040275s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.633219403 (499h7m29.504302438s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.666495797 (499h7m29.537578842s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.684666737 (499h7m29.555749772s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.686786163 (499h7m29.557869208s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.749653944 (499h7m29.620736989s) [noise=true , address=77.90.31.17:4790 , piece_id=NEM7KTUGPUJUFVO2AY33VYDZH5IVY75ZYSMHORG7SHLVOGNBAC3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.803202229 (499h7m29.674285274s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.812318928 (499h7m29.683401973s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884842114 (499h7m29.755925159s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884869236 (499h7m29.755952281s) [seg_pos={0 0}, upload=438077]: Segment reached optimal threshold of 14 pieces. Dec 26 10:36:29.885130354 (499h7m29.756213399s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885139761 (499h7m29.756222806s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885215545 (499h7m29.75629859s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885232007 (499h7m29.756315042s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885403694 (499h7m29.756486729s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection Dec 26 10:36:29.885410206 (499h7m29.756493242s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885477505 (499h7m29.75656055s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885486602 (499h7m29.756569647s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885509506 (499h7m29.756592541s) [seg_pos={0 0}, upload=438077]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:36:29.885594277 (499h7m29.756677322s) [upload=438077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:29.885600188 (499h7m29.756683223s) [upload=438077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:30.204831215 (499h7m30.07591425s) [upload=438077]: Done uploading segments... Dec 26 10:36:30.204857135 (499h7m30.07594017s) [upload=438077]: Upload finished. err: <nil> Duration: 31.494992744s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X..................X.......XXX......X............X........] duration: 40.56813132s trace: https://jaeger.dev.storj.io/trace/689c45a79c742610 FAIL (2025-12-26 10:44:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 10:36:27.577470372 (499h7m27.448553417s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.577672456 (499h7m27.448755491s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578119137 (499h7m27.449202172s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578562491 (499h7m27.449645536s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578686798 (499h7m27.449769843s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578965638 (499h7m27.450048684s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579314914 (499h7m27.450397959s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579456253 (499h7m27.450539328s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579866264 (499h7m27.450949299s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580237681 (499h7m27.451320716s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580270013 (499h7m27.451353048s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580573590 (499h7m27.451656635s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580865527 (499h7m27.451948562s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.581836275 (499h7m27.45291931s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.583318146 (499h7m27.454401191s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.585237882 (499h7m27.456320937s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:28.960965352 (499h7m28.832048387s) [upload=438077]: Next returned nil segment Dec 26 10:36:28.960982154 (499h7m28.832065199s) [upload=438077]: Waiting for error group managing segments... Dec 26 10:36:29.281740992 (499h7m29.152824037s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.457737227 (499h7m29.328820262s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.468672399 (499h7m29.339755434s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.474132455 (499h7m29.34521549s) [noise=true , address=145.79.227.49:4354 , piece_id=WXFH2GU2GMTNYG2FC3UEORTY2LCBUOYB3WXORXZ4RTSZZJFGIDCQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.563457636 (499h7m29.434540681s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.600957240 (499h7m29.472040275s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.633219403 (499h7m29.504302438s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.666495797 (499h7m29.537578842s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.684666737 (499h7m29.555749772s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.686786163 (499h7m29.557869208s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.749653944 (499h7m29.620736989s) [noise=true , address=77.90.31.17:4790 , piece_id=NEM7KTUGPUJUFVO2AY33VYDZH5IVY75ZYSMHORG7SHLVOGNBAC3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.803202229 (499h7m29.674285274s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.812318928 (499h7m29.683401973s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884842114 (499h7m29.755925159s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884869236 (499h7m29.755952281s) [seg_pos={0 0}, upload=438077]: Segment reached optimal threshold of 14 pieces. Dec 26 10:36:29.885130354 (499h7m29.756213399s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885139761 (499h7m29.756222806s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885215545 (499h7m29.75629859s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885232007 (499h7m29.756315042s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885403694 (499h7m29.756486729s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection Dec 26 10:36:29.885410206 (499h7m29.756493242s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885477505 (499h7m29.75656055s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885486602 (499h7m29.756569647s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885509506 (499h7m29.756592541s) [seg_pos={0 0}, upload=438077]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:36:29.885594277 (499h7m29.756677322s) [upload=438077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:29.885600188 (499h7m29.756683223s) [upload=438077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:30.204831215 (499h7m30.07591425s) [upload=438077]: Done uploading segments... Dec 26 10:36:30.204857135 (499h7m30.07594017s) [upload=438077]: Upload finished. err: <nil> Duration: 40.568148342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXX.X.X..XX...X.X.XX....XX.......X.X.....X..X....X..X..X....] duration: 6.025698164s trace: https://jaeger.dev.storj.io/trace/4338b564d5adc13a FAIL (2025-12-26 10:43:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ) context deadline exceeded ======LOGS====== Dec 26 10:40:43.709056067 (499h11m43.580139112s) [upload=438134]: Starting upload Dec 26 10:40:43.709110280 (499h11m43.580193325s) [upload=438134]: Done starting upload Dec 26 10:40:43.709167779 (499h11m43.580250815s) [upload=438134]: Uploading segments... Dec 26 10:40:43.709289561 (499h11m43.580372606s) [upload=438134]: Got next segment. Inline: false Dec 26 10:40:43.709316532 (499h11m43.580399568s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:40:43.709321101 (499h11m43.580404146s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:40:43.955318361 (499h11m43.826401406s) [seg_pos={0 0}, upload=438134]: Begin upload segment... Dec 26 10:40:43.955375580 (499h11m43.826458625s) [seg_pos={0 0}, upload=438134]: Done begin upload segment. Dec 26 10:40:43.955413943 (499h11m43.826496988s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.955829925 (499h11m43.82691299s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956269121 (499h11m43.827352156s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956570074 (499h11m43.827653109s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956603287 (499h11m43.827686342s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957175627 (499h11m43.828258692s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957403531 (499h11m43.828486576s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957731555 (499h11m43.8288146s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957757996 (499h11m43.828841041s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957852045 (499h11m43.82893509s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958244603 (499h11m43.829327648s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958689861 (499h11m43.829772906s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958798417 (499h11m43.829881452s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.959579504 (499h11m43.830662549s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960180179 (499h11m43.831263224s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960745525 (499h11m43.83182857s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:45.074144120 (499h11m44.945227155s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 10:40:45.670947462 (499h11m45.542030507s) [upload=438134]: Next returned nil segment Dec 26 10:40:45.670960177 (499h11m45.542043212s) [upload=438134]: Waiting for error group managing segments... Dec 26 10:40:45.954644609 (499h11m45.825727654s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.959526224 (499h11m45.830609259s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.964256672 (499h11m45.835339707s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.971375917 (499h11m45.842458952s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.972293225 (499h11m45.84337627s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.983096866 (499h11m45.854179912s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.985743005 (499h11m45.85682605s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986187622 (499h11m45.857270677s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986484257 (499h11m45.857567302s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.284280851 (499h11m46.155363886s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.326738467 (499h11m46.197821512s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.482782160 (499h11m46.353865205s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153035854 (499h11m47.024118889s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153057365 (499h11m47.0241404s) [seg_pos={0 0}, upload=438134]: Segment reached optimal threshold of 13 pieces. Dec 26 10:40:47.153218621 (499h11m47.024301657s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 10:40:47.153225585 (499h11m47.02430862s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153561755 (499h11m47.0246448s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection Dec 26 10:40:47.153583056 (499h11m47.024666101s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153617862 (499h11m47.024700907s) [seg_pos={0 0}, upload=438134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:40:47.153841508 (499h11m47.024924553s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:40:47.153849653 (499h11m47.024932688s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:40:47.437182455 (499h11m47.3082655s) [upload=438134]: Done uploading segments... Dec 26 10:40:47.437201211 (499h11m47.308284246s) [upload=438134]: Upload finished. err: <nil> Duration: 6.025722851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X..................X.......XXX......X............X.......] duration: 56.185161154s trace: https://jaeger.dev.storj.io/trace/689c45a79c742610 FAIL (2025-12-26 10:43:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 10:36:27.577470372 (499h7m27.448553417s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.577672456 (499h7m27.448755491s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578119137 (499h7m27.449202172s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578562491 (499h7m27.449645536s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578686798 (499h7m27.449769843s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578965638 (499h7m27.450048684s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579314914 (499h7m27.450397959s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579456253 (499h7m27.450539328s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579866264 (499h7m27.450949299s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580237681 (499h7m27.451320716s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580270013 (499h7m27.451353048s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580573590 (499h7m27.451656635s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580865527 (499h7m27.451948562s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.581836275 (499h7m27.45291931s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.583318146 (499h7m27.454401191s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.585237882 (499h7m27.456320937s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:28.960965352 (499h7m28.832048387s) [upload=438077]: Next returned nil segment Dec 26 10:36:28.960982154 (499h7m28.832065199s) [upload=438077]: Waiting for error group managing segments... Dec 26 10:36:29.281740992 (499h7m29.152824037s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.457737227 (499h7m29.328820262s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.468672399 (499h7m29.339755434s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.474132455 (499h7m29.34521549s) [noise=true , address=145.79.227.49:4354 , piece_id=WXFH2GU2GMTNYG2FC3UEORTY2LCBUOYB3WXORXZ4RTSZZJFGIDCQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.563457636 (499h7m29.434540681s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.600957240 (499h7m29.472040275s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.633219403 (499h7m29.504302438s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.666495797 (499h7m29.537578842s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.684666737 (499h7m29.555749772s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.686786163 (499h7m29.557869208s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.749653944 (499h7m29.620736989s) [noise=true , address=77.90.31.17:4790 , piece_id=NEM7KTUGPUJUFVO2AY33VYDZH5IVY75ZYSMHORG7SHLVOGNBAC3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.803202229 (499h7m29.674285274s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.812318928 (499h7m29.683401973s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884842114 (499h7m29.755925159s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884869236 (499h7m29.755952281s) [seg_pos={0 0}, upload=438077]: Segment reached optimal threshold of 14 pieces. Dec 26 10:36:29.885130354 (499h7m29.756213399s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885139761 (499h7m29.756222806s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885215545 (499h7m29.75629859s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885232007 (499h7m29.756315042s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885403694 (499h7m29.756486729s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection Dec 26 10:36:29.885410206 (499h7m29.756493242s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885477505 (499h7m29.75656055s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885486602 (499h7m29.756569647s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885509506 (499h7m29.756592541s) [seg_pos={0 0}, upload=438077]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:36:29.885594277 (499h7m29.756677322s) [upload=438077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:29.885600188 (499h7m29.756683223s) [upload=438077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:30.204831215 (499h7m30.07591425s) [upload=438077]: Done uploading segments... Dec 26 10:36:30.204857135 (499h7m30.07594017s) [upload=438077]: Upload finished. err: <nil> Duration: 56.18518596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X....X..X..X...] duration: 6.117789299s trace: https://jaeger.dev.storj.io/trace/4338b564d5adc13a FAIL (2025-12-26 10:43:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ) context deadline exceeded ======LOGS====== Dec 26 10:40:43.709056067 (499h11m43.580139112s) [upload=438134]: Starting upload Dec 26 10:40:43.709110280 (499h11m43.580193325s) [upload=438134]: Done starting upload Dec 26 10:40:43.709167779 (499h11m43.580250815s) [upload=438134]: Uploading segments... Dec 26 10:40:43.709289561 (499h11m43.580372606s) [upload=438134]: Got next segment. Inline: false Dec 26 10:40:43.709316532 (499h11m43.580399568s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:40:43.709321101 (499h11m43.580404146s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:40:43.955318361 (499h11m43.826401406s) [seg_pos={0 0}, upload=438134]: Begin upload segment... Dec 26 10:40:43.955375580 (499h11m43.826458625s) [seg_pos={0 0}, upload=438134]: Done begin upload segment. Dec 26 10:40:43.955413943 (499h11m43.826496988s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.955829925 (499h11m43.82691299s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956269121 (499h11m43.827352156s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956570074 (499h11m43.827653109s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956603287 (499h11m43.827686342s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957175627 (499h11m43.828258692s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957403531 (499h11m43.828486576s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957731555 (499h11m43.8288146s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957757996 (499h11m43.828841041s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957852045 (499h11m43.82893509s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958244603 (499h11m43.829327648s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958689861 (499h11m43.829772906s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958798417 (499h11m43.829881452s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.959579504 (499h11m43.830662549s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960180179 (499h11m43.831263224s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960745525 (499h11m43.83182857s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:45.074144120 (499h11m44.945227155s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 10:40:45.670947462 (499h11m45.542030507s) [upload=438134]: Next returned nil segment Dec 26 10:40:45.670960177 (499h11m45.542043212s) [upload=438134]: Waiting for error group managing segments... Dec 26 10:40:45.954644609 (499h11m45.825727654s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.959526224 (499h11m45.830609259s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.964256672 (499h11m45.835339707s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.971375917 (499h11m45.842458952s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.972293225 (499h11m45.84337627s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.983096866 (499h11m45.854179912s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.985743005 (499h11m45.85682605s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986187622 (499h11m45.857270677s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986484257 (499h11m45.857567302s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.284280851 (499h11m46.155363886s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.326738467 (499h11m46.197821512s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.482782160 (499h11m46.353865205s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153035854 (499h11m47.024118889s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153057365 (499h11m47.0241404s) [seg_pos={0 0}, upload=438134]: Segment reached optimal threshold of 13 pieces. Dec 26 10:40:47.153218621 (499h11m47.024301657s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 10:40:47.153225585 (499h11m47.02430862s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153561755 (499h11m47.0246448s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection Dec 26 10:40:47.153583056 (499h11m47.024666101s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153617862 (499h11m47.024700907s) [seg_pos={0 0}, upload=438134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:40:47.153841508 (499h11m47.024924553s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:40:47.153849653 (499h11m47.024932688s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:40:47.437182455 (499h11m47.3082655s) [upload=438134]: Done uploading segments... Dec 26 10:40:47.437201211 (499h11m47.308284246s) [upload=438134]: Upload finished. err: <nil> Duration: 6.117818035s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X..................X.......XXX......X............X......] duration: 44.590371131s trace: https://jaeger.dev.storj.io/trace/689c45a79c742610 FAIL (2025-12-26 10:42:24 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 10:36:27.577470372 (499h7m27.448553417s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.577672456 (499h7m27.448755491s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578119137 (499h7m27.449202172s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578562491 (499h7m27.449645536s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578686798 (499h7m27.449769843s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578965638 (499h7m27.450048684s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579314914 (499h7m27.450397959s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579456253 (499h7m27.450539328s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579866264 (499h7m27.450949299s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580237681 (499h7m27.451320716s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580270013 (499h7m27.451353048s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580573590 (499h7m27.451656635s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580865527 (499h7m27.451948562s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.581836275 (499h7m27.45291931s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.583318146 (499h7m27.454401191s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.585237882 (499h7m27.456320937s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:28.960965352 (499h7m28.832048387s) [upload=438077]: Next returned nil segment Dec 26 10:36:28.960982154 (499h7m28.832065199s) [upload=438077]: Waiting for error group managing segments... Dec 26 10:36:29.281740992 (499h7m29.152824037s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.457737227 (499h7m29.328820262s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.468672399 (499h7m29.339755434s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.474132455 (499h7m29.34521549s) [noise=true , address=145.79.227.49:4354 , piece_id=WXFH2GU2GMTNYG2FC3UEORTY2LCBUOYB3WXORXZ4RTSZZJFGIDCQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.563457636 (499h7m29.434540681s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.600957240 (499h7m29.472040275s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.633219403 (499h7m29.504302438s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.666495797 (499h7m29.537578842s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.684666737 (499h7m29.555749772s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.686786163 (499h7m29.557869208s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.749653944 (499h7m29.620736989s) [noise=true , address=77.90.31.17:4790 , piece_id=NEM7KTUGPUJUFVO2AY33VYDZH5IVY75ZYSMHORG7SHLVOGNBAC3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.803202229 (499h7m29.674285274s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.812318928 (499h7m29.683401973s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884842114 (499h7m29.755925159s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884869236 (499h7m29.755952281s) [seg_pos={0 0}, upload=438077]: Segment reached optimal threshold of 14 pieces. Dec 26 10:36:29.885130354 (499h7m29.756213399s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885139761 (499h7m29.756222806s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885215545 (499h7m29.75629859s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885232007 (499h7m29.756315042s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885403694 (499h7m29.756486729s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection Dec 26 10:36:29.885410206 (499h7m29.756493242s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885477505 (499h7m29.75656055s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885486602 (499h7m29.756569647s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885509506 (499h7m29.756592541s) [seg_pos={0 0}, upload=438077]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:36:29.885594277 (499h7m29.756677322s) [upload=438077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:29.885600188 (499h7m29.756683223s) [upload=438077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:30.204831215 (499h7m30.07591425s) [upload=438077]: Done uploading segments... Dec 26 10:36:30.204857135 (499h7m30.07594017s) [upload=438077]: Upload finished. err: <nil> Duration: 44.590385898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X....X..X..X..] duration: 6.018345011s trace: https://jaeger.dev.storj.io/trace/4338b564d5adc13a FAIL (2025-12-26 10:42:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ) context deadline exceeded ======LOGS====== Dec 26 10:40:43.709056067 (499h11m43.580139112s) [upload=438134]: Starting upload Dec 26 10:40:43.709110280 (499h11m43.580193325s) [upload=438134]: Done starting upload Dec 26 10:40:43.709167779 (499h11m43.580250815s) [upload=438134]: Uploading segments... Dec 26 10:40:43.709289561 (499h11m43.580372606s) [upload=438134]: Got next segment. Inline: false Dec 26 10:40:43.709316532 (499h11m43.580399568s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:40:43.709321101 (499h11m43.580404146s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:40:43.955318361 (499h11m43.826401406s) [seg_pos={0 0}, upload=438134]: Begin upload segment... Dec 26 10:40:43.955375580 (499h11m43.826458625s) [seg_pos={0 0}, upload=438134]: Done begin upload segment. Dec 26 10:40:43.955413943 (499h11m43.826496988s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.955829925 (499h11m43.82691299s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956269121 (499h11m43.827352156s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956570074 (499h11m43.827653109s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956603287 (499h11m43.827686342s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957175627 (499h11m43.828258692s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957403531 (499h11m43.828486576s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957731555 (499h11m43.8288146s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957757996 (499h11m43.828841041s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957852045 (499h11m43.82893509s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958244603 (499h11m43.829327648s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958689861 (499h11m43.829772906s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958798417 (499h11m43.829881452s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.959579504 (499h11m43.830662549s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960180179 (499h11m43.831263224s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960745525 (499h11m43.83182857s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:45.074144120 (499h11m44.945227155s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 10:40:45.670947462 (499h11m45.542030507s) [upload=438134]: Next returned nil segment Dec 26 10:40:45.670960177 (499h11m45.542043212s) [upload=438134]: Waiting for error group managing segments... Dec 26 10:40:45.954644609 (499h11m45.825727654s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.959526224 (499h11m45.830609259s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.964256672 (499h11m45.835339707s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.971375917 (499h11m45.842458952s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.972293225 (499h11m45.84337627s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.983096866 (499h11m45.854179912s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.985743005 (499h11m45.85682605s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986187622 (499h11m45.857270677s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986484257 (499h11m45.857567302s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.284280851 (499h11m46.155363886s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.326738467 (499h11m46.197821512s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.482782160 (499h11m46.353865205s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153035854 (499h11m47.024118889s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153057365 (499h11m47.0241404s) [seg_pos={0 0}, upload=438134]: Segment reached optimal threshold of 13 pieces. Dec 26 10:40:47.153218621 (499h11m47.024301657s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 10:40:47.153225585 (499h11m47.02430862s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153561755 (499h11m47.0246448s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection Dec 26 10:40:47.153583056 (499h11m47.024666101s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153617862 (499h11m47.024700907s) [seg_pos={0 0}, upload=438134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:40:47.153841508 (499h11m47.024924553s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:40:47.153849653 (499h11m47.024932688s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:40:47.437182455 (499h11m47.3082655s) [upload=438134]: Done uploading segments... Dec 26 10:40:47.437201211 (499h11m47.308284246s) [upload=438134]: Upload finished. err: <nil> Duration: 6.018372274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X....X..X..X.] duration: 5.972193161s trace: https://jaeger.dev.storj.io/trace/4338b564d5adc13a FAIL (2025-12-26 10:41:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ) context deadline exceeded ======LOGS====== Dec 26 10:40:43.709056067 (499h11m43.580139112s) [upload=438134]: Starting upload Dec 26 10:40:43.709110280 (499h11m43.580193325s) [upload=438134]: Done starting upload Dec 26 10:40:43.709167779 (499h11m43.580250815s) [upload=438134]: Uploading segments... Dec 26 10:40:43.709289561 (499h11m43.580372606s) [upload=438134]: Got next segment. Inline: false Dec 26 10:40:43.709316532 (499h11m43.580399568s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:40:43.709321101 (499h11m43.580404146s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:40:43.955318361 (499h11m43.826401406s) [seg_pos={0 0}, upload=438134]: Begin upload segment... Dec 26 10:40:43.955375580 (499h11m43.826458625s) [seg_pos={0 0}, upload=438134]: Done begin upload segment. Dec 26 10:40:43.955413943 (499h11m43.826496988s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.955829925 (499h11m43.82691299s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956269121 (499h11m43.827352156s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956570074 (499h11m43.827653109s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956603287 (499h11m43.827686342s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957175627 (499h11m43.828258692s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957403531 (499h11m43.828486576s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957731555 (499h11m43.8288146s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957757996 (499h11m43.828841041s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957852045 (499h11m43.82893509s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958244603 (499h11m43.829327648s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958689861 (499h11m43.829772906s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958798417 (499h11m43.829881452s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.959579504 (499h11m43.830662549s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960180179 (499h11m43.831263224s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960745525 (499h11m43.83182857s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:45.074144120 (499h11m44.945227155s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 10:40:45.670947462 (499h11m45.542030507s) [upload=438134]: Next returned nil segment Dec 26 10:40:45.670960177 (499h11m45.542043212s) [upload=438134]: Waiting for error group managing segments... Dec 26 10:40:45.954644609 (499h11m45.825727654s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.959526224 (499h11m45.830609259s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.964256672 (499h11m45.835339707s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.971375917 (499h11m45.842458952s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.972293225 (499h11m45.84337627s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.983096866 (499h11m45.854179912s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.985743005 (499h11m45.85682605s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986187622 (499h11m45.857270677s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986484257 (499h11m45.857567302s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.284280851 (499h11m46.155363886s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.326738467 (499h11m46.197821512s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.482782160 (499h11m46.353865205s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153035854 (499h11m47.024118889s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153057365 (499h11m47.0241404s) [seg_pos={0 0}, upload=438134]: Segment reached optimal threshold of 13 pieces. Dec 26 10:40:47.153218621 (499h11m47.024301657s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 10:40:47.153225585 (499h11m47.02430862s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153561755 (499h11m47.0246448s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection Dec 26 10:40:47.153583056 (499h11m47.024666101s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153617862 (499h11m47.024700907s) [seg_pos={0 0}, upload=438134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:40:47.153841508 (499h11m47.024924553s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:40:47.153849653 (499h11m47.024932688s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:40:47.437182455 (499h11m47.3082655s) [upload=438134]: Done uploading segments... Dec 26 10:40:47.437201211 (499h11m47.308284246s) [upload=438134]: Upload finished. err: <nil> Duration: 5.972232386s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X..................X.......XXX......X............X.....] duration: 51.830161814s trace: https://jaeger.dev.storj.io/trace/689c45a79c742610 FAIL (2025-12-26 10:41:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 10:36:27.577470372 (499h7m27.448553417s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.577672456 (499h7m27.448755491s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578119137 (499h7m27.449202172s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578562491 (499h7m27.449645536s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578686798 (499h7m27.449769843s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578965638 (499h7m27.450048684s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579314914 (499h7m27.450397959s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579456253 (499h7m27.450539328s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579866264 (499h7m27.450949299s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580237681 (499h7m27.451320716s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580270013 (499h7m27.451353048s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580573590 (499h7m27.451656635s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580865527 (499h7m27.451948562s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.581836275 (499h7m27.45291931s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.583318146 (499h7m27.454401191s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.585237882 (499h7m27.456320937s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:28.960965352 (499h7m28.832048387s) [upload=438077]: Next returned nil segment Dec 26 10:36:28.960982154 (499h7m28.832065199s) [upload=438077]: Waiting for error group managing segments... Dec 26 10:36:29.281740992 (499h7m29.152824037s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.457737227 (499h7m29.328820262s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.468672399 (499h7m29.339755434s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.474132455 (499h7m29.34521549s) [noise=true , address=145.79.227.49:4354 , piece_id=WXFH2GU2GMTNYG2FC3UEORTY2LCBUOYB3WXORXZ4RTSZZJFGIDCQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.563457636 (499h7m29.434540681s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.600957240 (499h7m29.472040275s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.633219403 (499h7m29.504302438s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.666495797 (499h7m29.537578842s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.684666737 (499h7m29.555749772s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.686786163 (499h7m29.557869208s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.749653944 (499h7m29.620736989s) [noise=true , address=77.90.31.17:4790 , piece_id=NEM7KTUGPUJUFVO2AY33VYDZH5IVY75ZYSMHORG7SHLVOGNBAC3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.803202229 (499h7m29.674285274s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.812318928 (499h7m29.683401973s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884842114 (499h7m29.755925159s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884869236 (499h7m29.755952281s) [seg_pos={0 0}, upload=438077]: Segment reached optimal threshold of 14 pieces. Dec 26 10:36:29.885130354 (499h7m29.756213399s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885139761 (499h7m29.756222806s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885215545 (499h7m29.75629859s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885232007 (499h7m29.756315042s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885403694 (499h7m29.756486729s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection Dec 26 10:36:29.885410206 (499h7m29.756493242s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885477505 (499h7m29.75656055s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885486602 (499h7m29.756569647s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885509506 (499h7m29.756592541s) [seg_pos={0 0}, upload=438077]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:36:29.885594277 (499h7m29.756677322s) [upload=438077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:29.885600188 (499h7m29.756683223s) [upload=438077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:30.204831215 (499h7m30.07591425s) [upload=438077]: Done uploading segments... Dec 26 10:36:30.204857135 (499h7m30.07594017s) [upload=438077]: Upload finished. err: <nil> Duration: 51.830185498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X....X..X..X] duration: 30.003602084s trace: https://jaeger.dev.storj.io/trace/4338b564d5adc13a FAIL (2025-12-26 10:41:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ) context deadline exceeded ======LOGS====== Dec 26 10:40:43.709056067 (499h11m43.580139112s) [upload=438134]: Starting upload Dec 26 10:40:43.709110280 (499h11m43.580193325s) [upload=438134]: Done starting upload Dec 26 10:40:43.709167779 (499h11m43.580250815s) [upload=438134]: Uploading segments... Dec 26 10:40:43.709289561 (499h11m43.580372606s) [upload=438134]: Got next segment. Inline: false Dec 26 10:40:43.709316532 (499h11m43.580399568s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:40:43.709321101 (499h11m43.580404146s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:40:43.955318361 (499h11m43.826401406s) [seg_pos={0 0}, upload=438134]: Begin upload segment... Dec 26 10:40:43.955375580 (499h11m43.826458625s) [seg_pos={0 0}, upload=438134]: Done begin upload segment. Dec 26 10:40:43.955413943 (499h11m43.826496988s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.955829925 (499h11m43.82691299s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956269121 (499h11m43.827352156s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956570074 (499h11m43.827653109s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.956603287 (499h11m43.827686342s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957175627 (499h11m43.828258692s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957403531 (499h11m43.828486576s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957731555 (499h11m43.8288146s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957757996 (499h11m43.828841041s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.957852045 (499h11m43.82893509s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958244603 (499h11m43.829327648s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958689861 (499h11m43.829772906s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.958798417 (499h11m43.829881452s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.959579504 (499h11m43.830662549s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960180179 (499h11m43.831263224s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:43.960745525 (499h11m43.83182857s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Uploading piece... Dec 26 10:40:45.074144120 (499h11m44.945227155s) [noise=true , address=77.90.31.18:4805 , piece_id=GNEHIWZO2QRJHXZWWUN34P3OH36BOKJJXG24QU3PBOC4O53EC4IQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 10:40:45.670947462 (499h11m45.542030507s) [upload=438134]: Next returned nil segment Dec 26 10:40:45.670960177 (499h11m45.542043212s) [upload=438134]: Waiting for error group managing segments... Dec 26 10:40:45.954644609 (499h11m45.825727654s) [noise=true , address=145.79.227.6:4257 , piece_id=2LTHIP3VY3JYJSCG3XE5YB7HBHKAXLPWDTJ4HQLLLZV65NK4XI4Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.959526224 (499h11m45.830609259s) [noise=true , address=145.79.227.13:4804 , piece_id=FRBAPLNEKWIPXQ4K65BMC6T6RSQSRXG6EBWRZEQPHWY6WA4F7YFA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.964256672 (499h11m45.835339707s) [noise=true , address=145.79.227.37:4567 , piece_id=Y7MRLA3KINDR3FSQOMC2AB6MJFSRQFGUZ7OPDVPVVYRFCBZ4VMKA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.971375917 (499h11m45.842458952s) [noise=true , address=145.79.227.38:4979 , piece_id=WT27EQ4LSE7PR74IOY4DPCT7OYJ32BLKBG6RT3JBBUD7H4OZA5UQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.972293225 (499h11m45.84337627s) [noise=true , address=145.79.227.5:4223 , piece_id=Q6SQ6UD6OEYJVL36BGR3NBV3SVGJF6FCVEPYMG4M4OM7R7QOIYAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.983096866 (499h11m45.854179912s) [noise=true , address=145.79.227.11:4730 , piece_id=U4GXWF5SXBXO2NGU5IQBPCTY7EFZ64DE4QQMGUTKXLVDCIVAAB5Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.985743005 (499h11m45.85682605s) [noise=true , address=145.79.227.50:4375 , piece_id=MBWC2JSIYL7NG2VMVG5KRAEEA4IQQZ4KGVZKXQ3JMOVNYVG24Q3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986187622 (499h11m45.857270677s) [noise=true , address=145.79.227.29:4941 , piece_id=OVHFAB7QWZZ2QHJK2EKP7M3MDQLKAZZBYK6CCXWVRNLWSEU3NWRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:45.986484257 (499h11m45.857567302s) [noise=true , address=145.79.227.41:4086 , piece_id=CYKDOZGRGA75CBH36CBVGMEQ22GDD6UGWTJRRFXIGZLQ6E3MVKAQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.284280851 (499h11m46.155363886s) [noise=true , address=45.91.116.12:4225 , piece_id=GNAQ4YZOM45SWZYNGJBGW7OMUAGBZC4MIFWSAGF3GDKIX3AMQYRQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.326738467 (499h11m46.197821512s) [noise=true , address=45.91.116.7:4010 , piece_id=BHUHVZG7UTAC6TMILUJT2GPUPZJZGDGNNOPCADHUN3EE53PE3OUQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:46.482782160 (499h11m46.353865205s) [noise=true , address=77.90.31.10:4520 , piece_id=R4G7T7LSW75TAIKN6DIYIQLBCM4Z7LW2IJYTAPCYAIGDZ3SCIXGQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153035854 (499h11m47.024118889s) [noise=true , address=45.91.116.10:4081 , piece_id=YAJF6UCZLNHSEPWTPB4YU4MYKYFXWWMSY6H6DJ7J7AHLDGXRMDKQ, seg_pos={0 0}, upload=438134]: Done uploading piece. err: <nil> Dec 26 10:40:47.153057365 (499h11m47.0241404s) [seg_pos={0 0}, upload=438134]: Segment reached optimal threshold of 13 pieces. Dec 26 10:40:47.153218621 (499h11m47.024301657s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 10:40:47.153225585 (499h11m47.02430862s) [noise=true , address=145.79.227.30:4761 , piece_id=5NM66J3CJVUABP6Z4BF6HWOCQQ5NAMGO43F4HJLAOT7DVYYAXM3Q, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153561755 (499h11m47.0246448s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46414->45.91.116.9:4262: use of closed network connection Dec 26 10:40:47.153583056 (499h11m47.024666101s) [noise=true , address=45.91.116.9:4262 , piece_id=HNHF2HAPDUVCTAQE2IX43YBM5IJLWDVNLDUQBWYL3GOYKR4A2WWA, seg_pos={0 0}, upload=438134]: Long tail cancellation due to optimal threshold Dec 26 10:40:47.153617862 (499h11m47.024700907s) [seg_pos={0 0}, upload=438134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:40:47.153841508 (499h11m47.024924553s) [upload=438134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:40:47.153849653 (499h11m47.024932688s) [upload=438134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:40:47.437182455 (499h11m47.3082655s) [upload=438134]: Done uploading segments... Dec 26 10:40:47.437201211 (499h11m47.308284246s) [upload=438134]: Upload finished. err: <nil> Duration: 30.003621641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X..................X.......XXX......X............X....] duration: 41.673918481s trace: https://jaeger.dev.storj.io/trace/689c45a79c742610 FAIL (2025-12-26 10:40:25 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 10:36:27.577470372 (499h7m27.448553417s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.577672456 (499h7m27.448755491s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578119137 (499h7m27.449202172s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578562491 (499h7m27.449645536s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578686798 (499h7m27.449769843s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578965638 (499h7m27.450048684s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579314914 (499h7m27.450397959s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579456253 (499h7m27.450539328s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579866264 (499h7m27.450949299s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580237681 (499h7m27.451320716s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580270013 (499h7m27.451353048s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580573590 (499h7m27.451656635s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580865527 (499h7m27.451948562s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.581836275 (499h7m27.45291931s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.583318146 (499h7m27.454401191s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.585237882 (499h7m27.456320937s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:28.960965352 (499h7m28.832048387s) [upload=438077]: Next returned nil segment Dec 26 10:36:28.960982154 (499h7m28.832065199s) [upload=438077]: Waiting for error group managing segments... Dec 26 10:36:29.281740992 (499h7m29.152824037s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.457737227 (499h7m29.328820262s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.468672399 (499h7m29.339755434s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.474132455 (499h7m29.34521549s) [noise=true , address=145.79.227.49:4354 , piece_id=WXFH2GU2GMTNYG2FC3UEORTY2LCBUOYB3WXORXZ4RTSZZJFGIDCQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.563457636 (499h7m29.434540681s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.600957240 (499h7m29.472040275s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.633219403 (499h7m29.504302438s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.666495797 (499h7m29.537578842s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.684666737 (499h7m29.555749772s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.686786163 (499h7m29.557869208s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.749653944 (499h7m29.620736989s) [noise=true , address=77.90.31.17:4790 , piece_id=NEM7KTUGPUJUFVO2AY33VYDZH5IVY75ZYSMHORG7SHLVOGNBAC3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.803202229 (499h7m29.674285274s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.812318928 (499h7m29.683401973s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884842114 (499h7m29.755925159s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884869236 (499h7m29.755952281s) [seg_pos={0 0}, upload=438077]: Segment reached optimal threshold of 14 pieces. Dec 26 10:36:29.885130354 (499h7m29.756213399s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885139761 (499h7m29.756222806s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885215545 (499h7m29.75629859s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885232007 (499h7m29.756315042s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885403694 (499h7m29.756486729s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection Dec 26 10:36:29.885410206 (499h7m29.756493242s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885477505 (499h7m29.75656055s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885486602 (499h7m29.756569647s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885509506 (499h7m29.756592541s) [seg_pos={0 0}, upload=438077]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:36:29.885594277 (499h7m29.756677322s) [upload=438077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:29.885600188 (499h7m29.756683223s) [upload=438077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:30.204831215 (499h7m30.07591425s) [upload=438077]: Done uploading segments... Dec 26 10:36:30.204857135 (499h7m30.07594017s) [upload=438077]: Upload finished. err: <nil> Duration: 41.673934101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X....X..X..] duration: 6.035740014s trace: https://jaeger.dev.storj.io/trace/141842994327346a FAIL (2025-12-26 10:39:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 2ZPSHLW7H67HW2FGZBUEP22PK4UOI4JAO2S2W4GSJVJA4UEE5Q4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: FUNMMG6WE4L7J4AUQYB7PSY3YYJZDEDXJ3FB6GTFPPX3MA5BH37A) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: MBTWROP6VC5NEDGCGOMDO5YHZT4XBV6VB4DYK2ZZNBYZ4GYWQ3SQ) context deadline exceeded ======LOGS====== Dec 26 10:38:27.079698879 (499h9m26.950781914s) [upload=438103]: Starting upload Dec 26 10:38:27.079774232 (499h9m26.950857297s) [upload=438103]: Done starting upload Dec 26 10:38:27.079988740 (499h9m26.951071795s) [upload=438103]: Uploading segments... Dec 26 10:38:27.080181338 (499h9m26.951264393s) [upload=438103]: Got next segment. Inline: false Dec 26 10:38:27.080232705 (499h9m26.95131574s) [upload=438103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:38:27.080261350 (499h9m26.951344385s) [upload=438103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:38:27.330086498 (499h9m27.201169543s) [seg_pos={0 0}, upload=438103]: Begin upload segment... Dec 26 10:38:27.330199393 (499h9m27.201282428s) [seg_pos={0 0}, upload=438103]: Done begin upload segment. Dec 26 10:38:27.330249880 (499h9m27.201332915s) [noise=true , address=145.79.227.40:4049 , piece_id=DYJPTIWFYL7A436C37MLAFRK5PCCBM47INDONWE76IICW2NPTRJQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.330294614 (499h9m27.201377649s) [noise=true , address=45.91.116.6:4975 , piece_id=FUNMMG6WE4L7J4AUQYB7PSY3YYJZDEDXJ3FB6GTFPPX3MA5BH37A, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331004556 (499h9m27.202087601s) [noise=true , address=145.79.227.50:4373 , piece_id=NAEJZRBARCXKEIBNX64E2CX5ZSYMM4WBLECZLNYJHISI2DAFJKWQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331046246 (499h9m27.202129291s) [noise=true , address=77.90.31.15:4700 , piece_id=DYK5NYWJZWG2ZZLTREBRCTXGTEE4NF4N7KJ66XQEHDBCHPFXYF7Q, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331366646 (499h9m27.202449691s) [noise=true , address=45.91.116.14:4191 , piece_id=MBTWROP6VC5NEDGCGOMDO5YHZT4XBV6VB4DYK2ZZNBYZ4GYWQ3SQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331511201 (499h9m27.202594236s) [noise=true , address=145.79.227.46:4601 , piece_id=ATICMFFLNFXACFJ2SGDZLGNWNMCKNPJWIIIOYTDNYW7CQK3ZIVLQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331680804 (499h9m27.202763839s) [noise=true , address=145.79.227.19:4079 , piece_id=ORQVNVGTQNELGDVUYRLE4TWJQEIGT4WQAZJLAZW4FHLAAJHMWAPQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332009359 (499h9m27.203092394s) [noise=true , address=145.79.227.37:4566 , piece_id=65XH6S36X7RMAWYWV2P3DXUWUMIBRGENALUUZBAGRDT3F27IMLPQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332089493 (499h9m27.203172538s) [noise=true , address=77.90.31.17:4773 , piece_id=2ZPSHLW7H67HW2FGZBUEP22PK4UOI4JAO2S2W4GSJVJA4UEE5Q4A, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332434018 (499h9m27.203517063s) [noise=true , address=45.91.116.10:4084 , piece_id=XNAIKC7MG5WQRA7D5GI2FBHHFJTONNRDJGKPWZ4SP5CQ4JMKAHJQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332563254 (499h9m27.203646289s) [noise=true , address=77.90.31.21:4878 , piece_id=RMVAZUPTC7V56Q6DKI6F3AJIYZYIWYZRSOTUXRHX4QK2OD45754A, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332760550 (499h9m27.203843595s) [noise=true , address=145.79.227.38:4977 , piece_id=J53D4SED6KCB3W34C7FOH6LAYFKQ2MQWVGVPJHIYCMVFRKRVRV6Q, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.333153909 (499h9m27.204236944s) [noise=true , address=45.91.116.11:4120 , piece_id=KWROAOR6SI4A4YDU5GV2VBUNVOFVEFBLRZGC6V3ZNASLXDF3O5XA, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.333622662 (499h9m27.204705707s) [noise=true , address=145.79.227.45:4231 , piece_id=EINXZBENI4CB3JWEPKDVJLJBBMKQXDXG7WK2IMVQRSZ6M7UWPJTQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.334741302 (499h9m27.205824337s) [noise=true , address=145.79.227.29:4942 , piece_id=ZARICITVZ4XLQY2UES3XDE2QWFKUWQ4XOGB3VKNFMDJIXRWOY2HA, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.335774128 (499h9m27.206857163s) [noise=true , address=77.90.31.14:4661 , piece_id=DRDSXA24RUZEPYPURKK25G3EKU26RCGL6TSLE72TKSMTUGDFAK4Q, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:29.017307676 (499h9m28.888390721s) [upload=438103]: Next returned nil segment Dec 26 10:38:29.017326131 (499h9m28.888409176s) [upload=438103]: Waiting for error group managing segments... Dec 26 10:38:29.329231641 (499h9m29.200314676s) [noise=true , address=145.79.227.46:4601 , piece_id=ATICMFFLNFXACFJ2SGDZLGNWNMCKNPJWIIIOYTDNYW7CQK3ZIVLQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.341747402 (499h9m29.212830437s) [noise=true , address=145.79.227.19:4079 , piece_id=ORQVNVGTQNELGDVUYRLE4TWJQEIGT4WQAZJLAZW4FHLAAJHMWAPQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.347188032 (499h9m29.218271067s) [noise=true , address=145.79.227.38:4977 , piece_id=J53D4SED6KCB3W34C7FOH6LAYFKQ2MQWVGVPJHIYCMVFRKRVRV6Q, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.349464858 (499h9m29.220547893s) [noise=true , address=145.79.227.45:4231 , piece_id=EINXZBENI4CB3JWEPKDVJLJBBMKQXDXG7WK2IMVQRSZ6M7UWPJTQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.354409503 (499h9m29.225492548s) [noise=true , address=145.79.227.37:4566 , piece_id=65XH6S36X7RMAWYWV2P3DXUWUMIBRGENALUUZBAGRDT3F27IMLPQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.366223940 (499h9m29.237306975s) [noise=true , address=145.79.227.29:4942 , piece_id=ZARICITVZ4XLQY2UES3XDE2QWFKUWQ4XOGB3VKNFMDJIXRWOY2HA, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.368007266 (499h9m29.239090301s) [noise=true , address=145.79.227.50:4373 , piece_id=NAEJZRBARCXKEIBNX64E2CX5ZSYMM4WBLECZLNYJHISI2DAFJKWQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.549565463 (499h9m29.420648498s) [noise=true , address=45.91.116.6:4975 , piece_id=FUNMMG6WE4L7J4AUQYB7PSY3YYJZDEDXJ3FB6GTFPPX3MA5BH37A, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.583844445 (499h9m29.45492749s) [noise=true , address=145.79.227.40:4049 , piece_id=DYJPTIWFYL7A436C37MLAFRK5PCCBM47INDONWE76IICW2NPTRJQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.596823199 (499h9m29.467906234s) [noise=true , address=45.91.116.10:4084 , piece_id=XNAIKC7MG5WQRA7D5GI2FBHHFJTONNRDJGKPWZ4SP5CQ4JMKAHJQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.606372623 (499h9m29.477455658s) [noise=true , address=45.91.116.14:4191 , piece_id=MBTWROP6VC5NEDGCGOMDO5YHZT4XBV6VB4DYK2ZZNBYZ4GYWQ3SQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.631434625 (499h9m29.50251766s) [noise=true , address=77.90.31.17:4773 , piece_id=2ZPSHLW7H67HW2FGZBUEP22PK4UOI4JAO2S2W4GSJVJA4UEE5Q4A, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.634293859 (499h9m29.505376904s) [noise=true , address=77.90.31.15:4700 , piece_id=DYK5NYWJZWG2ZZLTREBRCTXGTEE4NF4N7KJ66XQEHDBCHPFXYF7Q, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.634308979 (499h9m29.505392014s) [seg_pos={0 0}, upload=438103]: Segment reached optimal threshold of 13 pieces. Dec 26 10:38:29.634577549 (499h9m29.505660594s) [noise=true , address=77.90.31.14:4661 , piece_id=DRDSXA24RUZEPYPURKK25G3EKU26RCGL6TSLE72TKSMTUGDFAK4Q, seg_pos={0 0}, upload=438103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 26 10:38:29.634587930 (499h9m29.505670975s) [noise=true , address=77.90.31.14:4661 , piece_id=DRDSXA24RUZEPYPURKK25G3EKU26RCGL6TSLE72TKSMTUGDFAK4Q, seg_pos={0 0}, upload=438103]: Long tail cancellation due to optimal threshold Dec 26 10:38:29.634642724 (499h9m29.505725759s) [noise=true , address=45.91.116.11:4120 , piece_id=KWROAOR6SI4A4YDU5GV2VBUNVOFVEFBLRZGC6V3ZNASLXDF3O5XA, seg_pos={0 0}, upload=438103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 10:38:29.634651420 (499h9m29.505734455s) [noise=true , address=45.91.116.11:4120 , piece_id=KWROAOR6SI4A4YDU5GV2VBUNVOFVEFBLRZGC6V3ZNASLXDF3O5XA, seg_pos={0 0}, upload=438103]: Long tail cancellation due to optimal threshold Dec 26 10:38:29.634862292 (499h9m29.505945327s) [noise=true , address=77.90.31.21:4878 , piece_id=RMVAZUPTC7V56Q6DKI6F3AJIYZYIWYZRSOTUXRHX4QK2OD45754A, seg_pos={0 0}, upload=438103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: read tcp 188.166.225.170:46338->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:38:29.634869225 (499h9m29.50595226s) [noise=true , address=77.90.31.21:4878 , piece_id=RMVAZUPTC7V56Q6DKI6F3AJIYZYIWYZRSOTUXRHX4QK2OD45754A, seg_pos={0 0}, upload=438103]: Long tail cancellation due to optimal threshold Dec 26 10:38:29.634888962 (499h9m29.505971997s) [seg_pos={0 0}, upload=438103]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:38:29.634955690 (499h9m29.506038725s) [upload=438103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:38:29.634960078 (499h9m29.506043113s) [upload=438103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:38:29.922159904 (499h9m29.793242949s) [upload=438103]: Done uploading segments... Dec 26 10:38:29.922219187 (499h9m29.793302222s) [upload=438103]: Upload finished. err: <nil> Duration: 6.035766255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X..................X.......XXX......X............X...] duration: 34.646714504s trace: https://jaeger.dev.storj.io/trace/689c45a79c742610 FAIL (2025-12-26 10:39:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 10:36:27.577470372 (499h7m27.448553417s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.577672456 (499h7m27.448755491s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578119137 (499h7m27.449202172s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578562491 (499h7m27.449645536s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578686798 (499h7m27.449769843s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578965638 (499h7m27.450048684s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579314914 (499h7m27.450397959s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579456253 (499h7m27.450539328s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579866264 (499h7m27.450949299s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580237681 (499h7m27.451320716s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580270013 (499h7m27.451353048s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580573590 (499h7m27.451656635s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580865527 (499h7m27.451948562s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.581836275 (499h7m27.45291931s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.583318146 (499h7m27.454401191s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.585237882 (499h7m27.456320937s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:28.960965352 (499h7m28.832048387s) [upload=438077]: Next returned nil segment Dec 26 10:36:28.960982154 (499h7m28.832065199s) [upload=438077]: Waiting for error group managing segments... Dec 26 10:36:29.281740992 (499h7m29.152824037s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.457737227 (499h7m29.328820262s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.468672399 (499h7m29.339755434s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.474132455 (499h7m29.34521549s) [noise=true , address=145.79.227.49:4354 , piece_id=WXFH2GU2GMTNYG2FC3UEORTY2LCBUOYB3WXORXZ4RTSZZJFGIDCQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.563457636 (499h7m29.434540681s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.600957240 (499h7m29.472040275s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.633219403 (499h7m29.504302438s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.666495797 (499h7m29.537578842s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.684666737 (499h7m29.555749772s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.686786163 (499h7m29.557869208s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.749653944 (499h7m29.620736989s) [noise=true , address=77.90.31.17:4790 , piece_id=NEM7KTUGPUJUFVO2AY33VYDZH5IVY75ZYSMHORG7SHLVOGNBAC3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.803202229 (499h7m29.674285274s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.812318928 (499h7m29.683401973s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884842114 (499h7m29.755925159s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884869236 (499h7m29.755952281s) [seg_pos={0 0}, upload=438077]: Segment reached optimal threshold of 14 pieces. Dec 26 10:36:29.885130354 (499h7m29.756213399s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885139761 (499h7m29.756222806s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885215545 (499h7m29.75629859s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885232007 (499h7m29.756315042s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885403694 (499h7m29.756486729s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection Dec 26 10:36:29.885410206 (499h7m29.756493242s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885477505 (499h7m29.75656055s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885486602 (499h7m29.756569647s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885509506 (499h7m29.756592541s) [seg_pos={0 0}, upload=438077]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:36:29.885594277 (499h7m29.756677322s) [upload=438077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:29.885600188 (499h7m29.756683223s) [upload=438077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:30.204831215 (499h7m30.07591425s) [upload=438077]: Done uploading segments... Dec 26 10:36:30.204857135 (499h7m30.07594017s) [upload=438077]: Upload finished. err: <nil> Duration: 34.646734962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X....X..X.] duration: 5.672551595s trace: https://jaeger.dev.storj.io/trace/141842994327346a FAIL (2025-12-26 10:39:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 2ZPSHLW7H67HW2FGZBUEP22PK4UOI4JAO2S2W4GSJVJA4UEE5Q4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: FUNMMG6WE4L7J4AUQYB7PSY3YYJZDEDXJ3FB6GTFPPX3MA5BH37A) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: MBTWROP6VC5NEDGCGOMDO5YHZT4XBV6VB4DYK2ZZNBYZ4GYWQ3SQ) context deadline exceeded ======LOGS====== Dec 26 10:38:27.079698879 (499h9m26.950781914s) [upload=438103]: Starting upload Dec 26 10:38:27.079774232 (499h9m26.950857297s) [upload=438103]: Done starting upload Dec 26 10:38:27.079988740 (499h9m26.951071795s) [upload=438103]: Uploading segments... Dec 26 10:38:27.080181338 (499h9m26.951264393s) [upload=438103]: Got next segment. Inline: false Dec 26 10:38:27.080232705 (499h9m26.95131574s) [upload=438103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:38:27.080261350 (499h9m26.951344385s) [upload=438103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:38:27.330086498 (499h9m27.201169543s) [seg_pos={0 0}, upload=438103]: Begin upload segment... Dec 26 10:38:27.330199393 (499h9m27.201282428s) [seg_pos={0 0}, upload=438103]: Done begin upload segment. Dec 26 10:38:27.330249880 (499h9m27.201332915s) [noise=true , address=145.79.227.40:4049 , piece_id=DYJPTIWFYL7A436C37MLAFRK5PCCBM47INDONWE76IICW2NPTRJQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.330294614 (499h9m27.201377649s) [noise=true , address=45.91.116.6:4975 , piece_id=FUNMMG6WE4L7J4AUQYB7PSY3YYJZDEDXJ3FB6GTFPPX3MA5BH37A, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331004556 (499h9m27.202087601s) [noise=true , address=145.79.227.50:4373 , piece_id=NAEJZRBARCXKEIBNX64E2CX5ZSYMM4WBLECZLNYJHISI2DAFJKWQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331046246 (499h9m27.202129291s) [noise=true , address=77.90.31.15:4700 , piece_id=DYK5NYWJZWG2ZZLTREBRCTXGTEE4NF4N7KJ66XQEHDBCHPFXYF7Q, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331366646 (499h9m27.202449691s) [noise=true , address=45.91.116.14:4191 , piece_id=MBTWROP6VC5NEDGCGOMDO5YHZT4XBV6VB4DYK2ZZNBYZ4GYWQ3SQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331511201 (499h9m27.202594236s) [noise=true , address=145.79.227.46:4601 , piece_id=ATICMFFLNFXACFJ2SGDZLGNWNMCKNPJWIIIOYTDNYW7CQK3ZIVLQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331680804 (499h9m27.202763839s) [noise=true , address=145.79.227.19:4079 , piece_id=ORQVNVGTQNELGDVUYRLE4TWJQEIGT4WQAZJLAZW4FHLAAJHMWAPQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332009359 (499h9m27.203092394s) [noise=true , address=145.79.227.37:4566 , piece_id=65XH6S36X7RMAWYWV2P3DXUWUMIBRGENALUUZBAGRDT3F27IMLPQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332089493 (499h9m27.203172538s) [noise=true , address=77.90.31.17:4773 , piece_id=2ZPSHLW7H67HW2FGZBUEP22PK4UOI4JAO2S2W4GSJVJA4UEE5Q4A, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332434018 (499h9m27.203517063s) [noise=true , address=45.91.116.10:4084 , piece_id=XNAIKC7MG5WQRA7D5GI2FBHHFJTONNRDJGKPWZ4SP5CQ4JMKAHJQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332563254 (499h9m27.203646289s) [noise=true , address=77.90.31.21:4878 , piece_id=RMVAZUPTC7V56Q6DKI6F3AJIYZYIWYZRSOTUXRHX4QK2OD45754A, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332760550 (499h9m27.203843595s) [noise=true , address=145.79.227.38:4977 , piece_id=J53D4SED6KCB3W34C7FOH6LAYFKQ2MQWVGVPJHIYCMVFRKRVRV6Q, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.333153909 (499h9m27.204236944s) [noise=true , address=45.91.116.11:4120 , piece_id=KWROAOR6SI4A4YDU5GV2VBUNVOFVEFBLRZGC6V3ZNASLXDF3O5XA, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.333622662 (499h9m27.204705707s) [noise=true , address=145.79.227.45:4231 , piece_id=EINXZBENI4CB3JWEPKDVJLJBBMKQXDXG7WK2IMVQRSZ6M7UWPJTQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.334741302 (499h9m27.205824337s) [noise=true , address=145.79.227.29:4942 , piece_id=ZARICITVZ4XLQY2UES3XDE2QWFKUWQ4XOGB3VKNFMDJIXRWOY2HA, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.335774128 (499h9m27.206857163s) [noise=true , address=77.90.31.14:4661 , piece_id=DRDSXA24RUZEPYPURKK25G3EKU26RCGL6TSLE72TKSMTUGDFAK4Q, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:29.017307676 (499h9m28.888390721s) [upload=438103]: Next returned nil segment Dec 26 10:38:29.017326131 (499h9m28.888409176s) [upload=438103]: Waiting for error group managing segments... Dec 26 10:38:29.329231641 (499h9m29.200314676s) [noise=true , address=145.79.227.46:4601 , piece_id=ATICMFFLNFXACFJ2SGDZLGNWNMCKNPJWIIIOYTDNYW7CQK3ZIVLQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.341747402 (499h9m29.212830437s) [noise=true , address=145.79.227.19:4079 , piece_id=ORQVNVGTQNELGDVUYRLE4TWJQEIGT4WQAZJLAZW4FHLAAJHMWAPQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.347188032 (499h9m29.218271067s) [noise=true , address=145.79.227.38:4977 , piece_id=J53D4SED6KCB3W34C7FOH6LAYFKQ2MQWVGVPJHIYCMVFRKRVRV6Q, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.349464858 (499h9m29.220547893s) [noise=true , address=145.79.227.45:4231 , piece_id=EINXZBENI4CB3JWEPKDVJLJBBMKQXDXG7WK2IMVQRSZ6M7UWPJTQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.354409503 (499h9m29.225492548s) [noise=true , address=145.79.227.37:4566 , piece_id=65XH6S36X7RMAWYWV2P3DXUWUMIBRGENALUUZBAGRDT3F27IMLPQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.366223940 (499h9m29.237306975s) [noise=true , address=145.79.227.29:4942 , piece_id=ZARICITVZ4XLQY2UES3XDE2QWFKUWQ4XOGB3VKNFMDJIXRWOY2HA, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.368007266 (499h9m29.239090301s) [noise=true , address=145.79.227.50:4373 , piece_id=NAEJZRBARCXKEIBNX64E2CX5ZSYMM4WBLECZLNYJHISI2DAFJKWQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.549565463 (499h9m29.420648498s) [noise=true , address=45.91.116.6:4975 , piece_id=FUNMMG6WE4L7J4AUQYB7PSY3YYJZDEDXJ3FB6GTFPPX3MA5BH37A, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.583844445 (499h9m29.45492749s) [noise=true , address=145.79.227.40:4049 , piece_id=DYJPTIWFYL7A436C37MLAFRK5PCCBM47INDONWE76IICW2NPTRJQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.596823199 (499h9m29.467906234s) [noise=true , address=45.91.116.10:4084 , piece_id=XNAIKC7MG5WQRA7D5GI2FBHHFJTONNRDJGKPWZ4SP5CQ4JMKAHJQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.606372623 (499h9m29.477455658s) [noise=true , address=45.91.116.14:4191 , piece_id=MBTWROP6VC5NEDGCGOMDO5YHZT4XBV6VB4DYK2ZZNBYZ4GYWQ3SQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.631434625 (499h9m29.50251766s) [noise=true , address=77.90.31.17:4773 , piece_id=2ZPSHLW7H67HW2FGZBUEP22PK4UOI4JAO2S2W4GSJVJA4UEE5Q4A, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.634293859 (499h9m29.505376904s) [noise=true , address=77.90.31.15:4700 , piece_id=DYK5NYWJZWG2ZZLTREBRCTXGTEE4NF4N7KJ66XQEHDBCHPFXYF7Q, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.634308979 (499h9m29.505392014s) [seg_pos={0 0}, upload=438103]: Segment reached optimal threshold of 13 pieces. Dec 26 10:38:29.634577549 (499h9m29.505660594s) [noise=true , address=77.90.31.14:4661 , piece_id=DRDSXA24RUZEPYPURKK25G3EKU26RCGL6TSLE72TKSMTUGDFAK4Q, seg_pos={0 0}, upload=438103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 26 10:38:29.634587930 (499h9m29.505670975s) [noise=true , address=77.90.31.14:4661 , piece_id=DRDSXA24RUZEPYPURKK25G3EKU26RCGL6TSLE72TKSMTUGDFAK4Q, seg_pos={0 0}, upload=438103]: Long tail cancellation due to optimal threshold Dec 26 10:38:29.634642724 (499h9m29.505725759s) [noise=true , address=45.91.116.11:4120 , piece_id=KWROAOR6SI4A4YDU5GV2VBUNVOFVEFBLRZGC6V3ZNASLXDF3O5XA, seg_pos={0 0}, upload=438103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 10:38:29.634651420 (499h9m29.505734455s) [noise=true , address=45.91.116.11:4120 , piece_id=KWROAOR6SI4A4YDU5GV2VBUNVOFVEFBLRZGC6V3ZNASLXDF3O5XA, seg_pos={0 0}, upload=438103]: Long tail cancellation due to optimal threshold Dec 26 10:38:29.634862292 (499h9m29.505945327s) [noise=true , address=77.90.31.21:4878 , piece_id=RMVAZUPTC7V56Q6DKI6F3AJIYZYIWYZRSOTUXRHX4QK2OD45754A, seg_pos={0 0}, upload=438103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: read tcp 188.166.225.170:46338->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:38:29.634869225 (499h9m29.50595226s) [noise=true , address=77.90.31.21:4878 , piece_id=RMVAZUPTC7V56Q6DKI6F3AJIYZYIWYZRSOTUXRHX4QK2OD45754A, seg_pos={0 0}, upload=438103]: Long tail cancellation due to optimal threshold Dec 26 10:38:29.634888962 (499h9m29.505971997s) [seg_pos={0 0}, upload=438103]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:38:29.634955690 (499h9m29.506038725s) [upload=438103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:38:29.634960078 (499h9m29.506043113s) [upload=438103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:38:29.922159904 (499h9m29.793242949s) [upload=438103]: Done uploading segments... Dec 26 10:38:29.922219187 (499h9m29.793302222s) [upload=438103]: Upload finished. err: <nil> Duration: 5.672578315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X..................X.......XXX......X............X..] duration: 43.087152881s trace: https://jaeger.dev.storj.io/trace/689c45a79c742610 FAIL (2025-12-26 10:39:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 10:36:27.577470372 (499h7m27.448553417s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.577672456 (499h7m27.448755491s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578119137 (499h7m27.449202172s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578562491 (499h7m27.449645536s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578686798 (499h7m27.449769843s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578965638 (499h7m27.450048684s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579314914 (499h7m27.450397959s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579456253 (499h7m27.450539328s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579866264 (499h7m27.450949299s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580237681 (499h7m27.451320716s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580270013 (499h7m27.451353048s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580573590 (499h7m27.451656635s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580865527 (499h7m27.451948562s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.581836275 (499h7m27.45291931s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.583318146 (499h7m27.454401191s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.585237882 (499h7m27.456320937s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:28.960965352 (499h7m28.832048387s) [upload=438077]: Next returned nil segment Dec 26 10:36:28.960982154 (499h7m28.832065199s) [upload=438077]: Waiting for error group managing segments... Dec 26 10:36:29.281740992 (499h7m29.152824037s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.457737227 (499h7m29.328820262s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.468672399 (499h7m29.339755434s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.474132455 (499h7m29.34521549s) [noise=true , address=145.79.227.49:4354 , piece_id=WXFH2GU2GMTNYG2FC3UEORTY2LCBUOYB3WXORXZ4RTSZZJFGIDCQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.563457636 (499h7m29.434540681s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.600957240 (499h7m29.472040275s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.633219403 (499h7m29.504302438s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.666495797 (499h7m29.537578842s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.684666737 (499h7m29.555749772s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.686786163 (499h7m29.557869208s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.749653944 (499h7m29.620736989s) [noise=true , address=77.90.31.17:4790 , piece_id=NEM7KTUGPUJUFVO2AY33VYDZH5IVY75ZYSMHORG7SHLVOGNBAC3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.803202229 (499h7m29.674285274s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.812318928 (499h7m29.683401973s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884842114 (499h7m29.755925159s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884869236 (499h7m29.755952281s) [seg_pos={0 0}, upload=438077]: Segment reached optimal threshold of 14 pieces. Dec 26 10:36:29.885130354 (499h7m29.756213399s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885139761 (499h7m29.756222806s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885215545 (499h7m29.75629859s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885232007 (499h7m29.756315042s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885403694 (499h7m29.756486729s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection Dec 26 10:36:29.885410206 (499h7m29.756493242s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885477505 (499h7m29.75656055s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885486602 (499h7m29.756569647s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885509506 (499h7m29.756592541s) [seg_pos={0 0}, upload=438077]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:36:29.885594277 (499h7m29.756677322s) [upload=438077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:29.885600188 (499h7m29.756683223s) [upload=438077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:30.204831215 (499h7m30.07591425s) [upload=438077]: Done uploading segments... Dec 26 10:36:30.204857135 (499h7m30.07594017s) [upload=438077]: Upload finished. err: <nil> Duration: 43.087167498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X....X..X] duration: 30.003376838s trace: https://jaeger.dev.storj.io/trace/141842994327346a FAIL (2025-12-26 10:38:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 2ZPSHLW7H67HW2FGZBUEP22PK4UOI4JAO2S2W4GSJVJA4UEE5Q4A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: FUNMMG6WE4L7J4AUQYB7PSY3YYJZDEDXJ3FB6GTFPPX3MA5BH37A) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: MBTWROP6VC5NEDGCGOMDO5YHZT4XBV6VB4DYK2ZZNBYZ4GYWQ3SQ) context deadline exceeded ======LOGS====== Dec 26 10:38:27.079698879 (499h9m26.950781914s) [upload=438103]: Starting upload Dec 26 10:38:27.079774232 (499h9m26.950857297s) [upload=438103]: Done starting upload Dec 26 10:38:27.079988740 (499h9m26.951071795s) [upload=438103]: Uploading segments... Dec 26 10:38:27.080181338 (499h9m26.951264393s) [upload=438103]: Got next segment. Inline: false Dec 26 10:38:27.080232705 (499h9m26.95131574s) [upload=438103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:38:27.080261350 (499h9m26.951344385s) [upload=438103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:38:27.330086498 (499h9m27.201169543s) [seg_pos={0 0}, upload=438103]: Begin upload segment... Dec 26 10:38:27.330199393 (499h9m27.201282428s) [seg_pos={0 0}, upload=438103]: Done begin upload segment. Dec 26 10:38:27.330249880 (499h9m27.201332915s) [noise=true , address=145.79.227.40:4049 , piece_id=DYJPTIWFYL7A436C37MLAFRK5PCCBM47INDONWE76IICW2NPTRJQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.330294614 (499h9m27.201377649s) [noise=true , address=45.91.116.6:4975 , piece_id=FUNMMG6WE4L7J4AUQYB7PSY3YYJZDEDXJ3FB6GTFPPX3MA5BH37A, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331004556 (499h9m27.202087601s) [noise=true , address=145.79.227.50:4373 , piece_id=NAEJZRBARCXKEIBNX64E2CX5ZSYMM4WBLECZLNYJHISI2DAFJKWQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331046246 (499h9m27.202129291s) [noise=true , address=77.90.31.15:4700 , piece_id=DYK5NYWJZWG2ZZLTREBRCTXGTEE4NF4N7KJ66XQEHDBCHPFXYF7Q, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331366646 (499h9m27.202449691s) [noise=true , address=45.91.116.14:4191 , piece_id=MBTWROP6VC5NEDGCGOMDO5YHZT4XBV6VB4DYK2ZZNBYZ4GYWQ3SQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331511201 (499h9m27.202594236s) [noise=true , address=145.79.227.46:4601 , piece_id=ATICMFFLNFXACFJ2SGDZLGNWNMCKNPJWIIIOYTDNYW7CQK3ZIVLQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.331680804 (499h9m27.202763839s) [noise=true , address=145.79.227.19:4079 , piece_id=ORQVNVGTQNELGDVUYRLE4TWJQEIGT4WQAZJLAZW4FHLAAJHMWAPQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332009359 (499h9m27.203092394s) [noise=true , address=145.79.227.37:4566 , piece_id=65XH6S36X7RMAWYWV2P3DXUWUMIBRGENALUUZBAGRDT3F27IMLPQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332089493 (499h9m27.203172538s) [noise=true , address=77.90.31.17:4773 , piece_id=2ZPSHLW7H67HW2FGZBUEP22PK4UOI4JAO2S2W4GSJVJA4UEE5Q4A, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332434018 (499h9m27.203517063s) [noise=true , address=45.91.116.10:4084 , piece_id=XNAIKC7MG5WQRA7D5GI2FBHHFJTONNRDJGKPWZ4SP5CQ4JMKAHJQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332563254 (499h9m27.203646289s) [noise=true , address=77.90.31.21:4878 , piece_id=RMVAZUPTC7V56Q6DKI6F3AJIYZYIWYZRSOTUXRHX4QK2OD45754A, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.332760550 (499h9m27.203843595s) [noise=true , address=145.79.227.38:4977 , piece_id=J53D4SED6KCB3W34C7FOH6LAYFKQ2MQWVGVPJHIYCMVFRKRVRV6Q, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.333153909 (499h9m27.204236944s) [noise=true , address=45.91.116.11:4120 , piece_id=KWROAOR6SI4A4YDU5GV2VBUNVOFVEFBLRZGC6V3ZNASLXDF3O5XA, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.333622662 (499h9m27.204705707s) [noise=true , address=145.79.227.45:4231 , piece_id=EINXZBENI4CB3JWEPKDVJLJBBMKQXDXG7WK2IMVQRSZ6M7UWPJTQ, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.334741302 (499h9m27.205824337s) [noise=true , address=145.79.227.29:4942 , piece_id=ZARICITVZ4XLQY2UES3XDE2QWFKUWQ4XOGB3VKNFMDJIXRWOY2HA, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:27.335774128 (499h9m27.206857163s) [noise=true , address=77.90.31.14:4661 , piece_id=DRDSXA24RUZEPYPURKK25G3EKU26RCGL6TSLE72TKSMTUGDFAK4Q, seg_pos={0 0}, upload=438103]: Uploading piece... Dec 26 10:38:29.017307676 (499h9m28.888390721s) [upload=438103]: Next returned nil segment Dec 26 10:38:29.017326131 (499h9m28.888409176s) [upload=438103]: Waiting for error group managing segments... Dec 26 10:38:29.329231641 (499h9m29.200314676s) [noise=true , address=145.79.227.46:4601 , piece_id=ATICMFFLNFXACFJ2SGDZLGNWNMCKNPJWIIIOYTDNYW7CQK3ZIVLQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.341747402 (499h9m29.212830437s) [noise=true , address=145.79.227.19:4079 , piece_id=ORQVNVGTQNELGDVUYRLE4TWJQEIGT4WQAZJLAZW4FHLAAJHMWAPQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.347188032 (499h9m29.218271067s) [noise=true , address=145.79.227.38:4977 , piece_id=J53D4SED6KCB3W34C7FOH6LAYFKQ2MQWVGVPJHIYCMVFRKRVRV6Q, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.349464858 (499h9m29.220547893s) [noise=true , address=145.79.227.45:4231 , piece_id=EINXZBENI4CB3JWEPKDVJLJBBMKQXDXG7WK2IMVQRSZ6M7UWPJTQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.354409503 (499h9m29.225492548s) [noise=true , address=145.79.227.37:4566 , piece_id=65XH6S36X7RMAWYWV2P3DXUWUMIBRGENALUUZBAGRDT3F27IMLPQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.366223940 (499h9m29.237306975s) [noise=true , address=145.79.227.29:4942 , piece_id=ZARICITVZ4XLQY2UES3XDE2QWFKUWQ4XOGB3VKNFMDJIXRWOY2HA, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.368007266 (499h9m29.239090301s) [noise=true , address=145.79.227.50:4373 , piece_id=NAEJZRBARCXKEIBNX64E2CX5ZSYMM4WBLECZLNYJHISI2DAFJKWQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.549565463 (499h9m29.420648498s) [noise=true , address=45.91.116.6:4975 , piece_id=FUNMMG6WE4L7J4AUQYB7PSY3YYJZDEDXJ3FB6GTFPPX3MA5BH37A, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.583844445 (499h9m29.45492749s) [noise=true , address=145.79.227.40:4049 , piece_id=DYJPTIWFYL7A436C37MLAFRK5PCCBM47INDONWE76IICW2NPTRJQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.596823199 (499h9m29.467906234s) [noise=true , address=45.91.116.10:4084 , piece_id=XNAIKC7MG5WQRA7D5GI2FBHHFJTONNRDJGKPWZ4SP5CQ4JMKAHJQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.606372623 (499h9m29.477455658s) [noise=true , address=45.91.116.14:4191 , piece_id=MBTWROP6VC5NEDGCGOMDO5YHZT4XBV6VB4DYK2ZZNBYZ4GYWQ3SQ, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.631434625 (499h9m29.50251766s) [noise=true , address=77.90.31.17:4773 , piece_id=2ZPSHLW7H67HW2FGZBUEP22PK4UOI4JAO2S2W4GSJVJA4UEE5Q4A, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.634293859 (499h9m29.505376904s) [noise=true , address=77.90.31.15:4700 , piece_id=DYK5NYWJZWG2ZZLTREBRCTXGTEE4NF4N7KJ66XQEHDBCHPFXYF7Q, seg_pos={0 0}, upload=438103]: Done uploading piece. err: <nil> Dec 26 10:38:29.634308979 (499h9m29.505392014s) [seg_pos={0 0}, upload=438103]: Segment reached optimal threshold of 13 pieces. Dec 26 10:38:29.634577549 (499h9m29.505660594s) [noise=true , address=77.90.31.14:4661 , piece_id=DRDSXA24RUZEPYPURKK25G3EKU26RCGL6TSLE72TKSMTUGDFAK4Q, seg_pos={0 0}, upload=438103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 26 10:38:29.634587930 (499h9m29.505670975s) [noise=true , address=77.90.31.14:4661 , piece_id=DRDSXA24RUZEPYPURKK25G3EKU26RCGL6TSLE72TKSMTUGDFAK4Q, seg_pos={0 0}, upload=438103]: Long tail cancellation due to optimal threshold Dec 26 10:38:29.634642724 (499h9m29.505725759s) [noise=true , address=45.91.116.11:4120 , piece_id=KWROAOR6SI4A4YDU5GV2VBUNVOFVEFBLRZGC6V3ZNASLXDF3O5XA, seg_pos={0 0}, upload=438103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 10:38:29.634651420 (499h9m29.505734455s) [noise=true , address=45.91.116.11:4120 , piece_id=KWROAOR6SI4A4YDU5GV2VBUNVOFVEFBLRZGC6V3ZNASLXDF3O5XA, seg_pos={0 0}, upload=438103]: Long tail cancellation due to optimal threshold Dec 26 10:38:29.634862292 (499h9m29.505945327s) [noise=true , address=77.90.31.21:4878 , piece_id=RMVAZUPTC7V56Q6DKI6F3AJIYZYIWYZRSOTUXRHX4QK2OD45754A, seg_pos={0 0}, upload=438103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: read tcp 188.166.225.170:46338->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:38:29.634869225 (499h9m29.50595226s) [noise=true , address=77.90.31.21:4878 , piece_id=RMVAZUPTC7V56Q6DKI6F3AJIYZYIWYZRSOTUXRHX4QK2OD45754A, seg_pos={0 0}, upload=438103]: Long tail cancellation due to optimal threshold Dec 26 10:38:29.634888962 (499h9m29.505971997s) [seg_pos={0 0}, upload=438103]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:38:29.634955690 (499h9m29.506038725s) [upload=438103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:38:29.634960078 (499h9m29.506043113s) [upload=438103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:38:29.922159904 (499h9m29.793242949s) [upload=438103]: Done uploading segments... Dec 26 10:38:29.922219187 (499h9m29.793302222s) [upload=438103]: Upload finished. err: <nil> Duration: 30.003401235s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X..................X.......XXX......X............X.] duration: 51.515010488s trace: https://jaeger.dev.storj.io/trace/689c45a79c742610 FAIL (2025-12-26 10:38:25 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 10:36:27.577470372 (499h7m27.448553417s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.577672456 (499h7m27.448755491s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578119137 (499h7m27.449202172s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578562491 (499h7m27.449645536s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578686798 (499h7m27.449769843s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578965638 (499h7m27.450048684s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579314914 (499h7m27.450397959s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579456253 (499h7m27.450539328s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579866264 (499h7m27.450949299s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580237681 (499h7m27.451320716s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580270013 (499h7m27.451353048s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580573590 (499h7m27.451656635s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580865527 (499h7m27.451948562s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.581836275 (499h7m27.45291931s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.583318146 (499h7m27.454401191s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.585237882 (499h7m27.456320937s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:28.960965352 (499h7m28.832048387s) [upload=438077]: Next returned nil segment Dec 26 10:36:28.960982154 (499h7m28.832065199s) [upload=438077]: Waiting for error group managing segments... Dec 26 10:36:29.281740992 (499h7m29.152824037s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.457737227 (499h7m29.328820262s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.468672399 (499h7m29.339755434s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.474132455 (499h7m29.34521549s) [noise=true , address=145.79.227.49:4354 , piece_id=WXFH2GU2GMTNYG2FC3UEORTY2LCBUOYB3WXORXZ4RTSZZJFGIDCQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.563457636 (499h7m29.434540681s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.600957240 (499h7m29.472040275s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.633219403 (499h7m29.504302438s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.666495797 (499h7m29.537578842s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.684666737 (499h7m29.555749772s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.686786163 (499h7m29.557869208s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.749653944 (499h7m29.620736989s) [noise=true , address=77.90.31.17:4790 , piece_id=NEM7KTUGPUJUFVO2AY33VYDZH5IVY75ZYSMHORG7SHLVOGNBAC3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.803202229 (499h7m29.674285274s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.812318928 (499h7m29.683401973s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884842114 (499h7m29.755925159s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884869236 (499h7m29.755952281s) [seg_pos={0 0}, upload=438077]: Segment reached optimal threshold of 14 pieces. Dec 26 10:36:29.885130354 (499h7m29.756213399s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885139761 (499h7m29.756222806s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885215545 (499h7m29.75629859s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885232007 (499h7m29.756315042s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885403694 (499h7m29.756486729s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection Dec 26 10:36:29.885410206 (499h7m29.756493242s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885477505 (499h7m29.75656055s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885486602 (499h7m29.756569647s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885509506 (499h7m29.756592541s) [seg_pos={0 0}, upload=438077]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:36:29.885594277 (499h7m29.756677322s) [upload=438077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:29.885600188 (499h7m29.756683223s) [upload=438077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:30.204831215 (499h7m30.07591425s) [upload=438077]: Done uploading segments... Dec 26 10:36:30.204857135 (499h7m30.07594017s) [upload=438077]: Upload finished. err: <nil> Duration: 51.51503722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X....X..] duration: 6.073896562s trace: https://jaeger.dev.storj.io/trace/43c0a796c19ad6f7 FAIL (2025-12-26 10:37:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 10:36:22.815974307 (499h7m22.687057352s) [upload=438074]: Starting upload Dec 26 10:36:22.816042687 (499h7m22.687125732s) [upload=438074]: Done starting upload Dec 26 10:36:22.816185108 (499h7m22.687268153s) [upload=438074]: Uploading segments... Dec 26 10:36:22.816287734 (499h7m22.687370769s) [upload=438074]: Got next segment. Inline: false Dec 26 10:36:22.816333871 (499h7m22.687416916s) [upload=438074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:36:22.816342008 (499h7m22.687425043s) [upload=438074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:36:23.061064477 (499h7m22.932147522s) [seg_pos={0 0}, upload=438074]: Begin upload segment... Dec 26 10:36:23.061116636 (499h7m22.932199681s) [seg_pos={0 0}, upload=438074]: Done begin upload segment. Dec 26 10:36:23.061151703 (499h7m22.932234748s) [noise=true , address=145.79.227.47:4265 , piece_id=BSTBADCISWBMHK7TM7TB5UYECIC5EU3CGVRQWKBWDYWRTIO7KBRA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.061284396 (499h7m22.932367441s) [noise=true , address=145.79.227.33:4870 , piece_id=2IYNQGXQESUEQDK2CXKX3RTI5S77DQCDT3OGIVYN4UTA2D6N46AQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.061943471 (499h7m22.933026506s) [noise=true , address=145.79.227.45:4230 , piece_id=AP3CZ4O4NLI4BHCK6F5QKW7VNA2D3HEL23ZYLTR5OBLMKI7IYHQQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.062468010 (499h7m22.933551055s) [noise=true , address=145.79.227.50:4374 , piece_id=CPNHLDRBRTB2UI4GM2Z6B4ZYO4GCGIUZBLLMFLDQCJC6SVHDB7VQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.062940319 (499h7m22.934023364s) [noise=true , address=45.91.116.4:4939 , piece_id=JDLO6ZIHDUJVPVXOICYFTSHZJZGMXWKMZAMB4SOJY6PYGQ2OUKVQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.063428349 (499h7m22.934511394s) [noise=true , address=145.79.227.48:4302 , piece_id=ZU7VPDLPFK5CPFOG565U5XDNOWDZK6CBLVO4P3YQZTYN3N4BFNPA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.063441884 (499h7m22.934524929s) [noise=true , address=145.79.227.49:4337 , piece_id=LEFMRDLXSVBQJP43BDIHQPM3JHPQJV5GX5L4DU7B4ED3OSES3YPA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.063496488 (499h7m22.934579524s) [noise=true , address=145.79.227.42:4122 , piece_id=C6WD3VRNXRHDISYCSNHGK2JIQXRPKSZYYNXT2MB3MMATNKIYRTTQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.063987173 (499h7m22.935070208s) [noise=true , address=145.79.227.36:4519 , piece_id=WT6JYHVJ2LVDBQGXM45MZAQSZLD5F2ZGHMOV3XKI6WFFH75FNKOQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.064459411 (499h7m22.935542456s) [noise=true , address=145.79.227.6:4259 , piece_id=WM4LUF4D2WGNBAWCKGGVYMSDKPKJGCTU7V3PPWK37GPJD2FXQRTQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.064971787 (499h7m22.936054822s) [noise=true , address=145.79.227.30:4762 , piece_id=G5HRN3RCGYZW5SXLEWC4PWIX7RALLPWF3Q5EMT7FE3E3NA4B3VEA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.065477159 (499h7m22.936560194s) [noise=true , address=145.79.227.43:4159 , piece_id=YVYPM4G7AI7K7AOFC36TDKHRXG3XGDCSJBA6BQRQ5AY72LR4ZZVA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.066065700 (499h7m22.937148745s) [noise=true , address=145.79.227.32:4835 , piece_id=UCKQRSYJQVNVT47KX2I7HP4VQGDIFWDFZIKMWI65TPK34LLET7RQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.066587363 (499h7m22.937670408s) [noise=true , address=77.90.31.10:4520 , piece_id=AFEZB7EPXTQVIY6FWPJ5F4R72Z667KKAMADKVLWBXUHJSJPBHA5A, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.068133508 (499h7m22.939216553s) [noise=true , address=45.91.116.7:4012 , piece_id=DEAO3Z74NO76YYJPBGZABQPISEPXVLRRYGFHN5FCGC5YZJKIM5TA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.068371291 (499h7m22.939454336s) [noise=true , address=77.90.31.11:4556 , piece_id=A7PJWV2ZCBFBGNXDY7XDXL4ZLTMJ4UDZXRKT5OXW6O43L7M6P6FA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:24.734655969 (499h7m24.605739004s) [upload=438074]: Next returned nil segment Dec 26 10:36:24.734670998 (499h7m24.605754033s) [upload=438074]: Waiting for error group managing segments... Dec 26 10:36:25.066064023 (499h7m24.937147059s) [noise=true , address=145.79.227.6:4259 , piece_id=WM4LUF4D2WGNBAWCKGGVYMSDKPKJGCTU7V3PPWK37GPJD2FXQRTQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.070619368 (499h7m24.941702403s) [noise=true , address=145.79.227.47:4265 , piece_id=BSTBADCISWBMHK7TM7TB5UYECIC5EU3CGVRQWKBWDYWRTIO7KBRA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.072315528 (499h7m24.943398573s) [noise=true , address=145.79.227.50:4374 , piece_id=CPNHLDRBRTB2UI4GM2Z6B4ZYO4GCGIUZBLLMFLDQCJC6SVHDB7VQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.072503976 (499h7m24.943587011s) [noise=true , address=145.79.227.49:4337 , piece_id=LEFMRDLXSVBQJP43BDIHQPM3JHPQJV5GX5L4DU7B4ED3OSES3YPA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.073316895 (499h7m24.94439993s) [noise=true , address=145.79.227.43:4159 , piece_id=YVYPM4G7AI7K7AOFC36TDKHRXG3XGDCSJBA6BQRQ5AY72LR4ZZVA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.075446830 (499h7m24.946529876s) [noise=true , address=145.79.227.45:4230 , piece_id=AP3CZ4O4NLI4BHCK6F5QKW7VNA2D3HEL23ZYLTR5OBLMKI7IYHQQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.076975842 (499h7m24.948058887s) [noise=true , address=145.79.227.48:4302 , piece_id=ZU7VPDLPFK5CPFOG565U5XDNOWDZK6CBLVO4P3YQZTYN3N4BFNPA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.088131574 (499h7m24.959214619s) [noise=true , address=145.79.227.42:4122 , piece_id=C6WD3VRNXRHDISYCSNHGK2JIQXRPKSZYYNXT2MB3MMATNKIYRTTQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.099031991 (499h7m24.970115036s) [noise=true , address=145.79.227.32:4835 , piece_id=UCKQRSYJQVNVT47KX2I7HP4VQGDIFWDFZIKMWI65TPK34LLET7RQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.100283463 (499h7m24.971366498s) [noise=true , address=145.79.227.36:4519 , piece_id=WT6JYHVJ2LVDBQGXM45MZAQSZLD5F2ZGHMOV3XKI6WFFH75FNKOQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.317502444 (499h7m25.188585489s) [noise=true , address=77.90.31.11:4556 , piece_id=A7PJWV2ZCBFBGNXDY7XDXL4ZLTMJ4UDZXRKT5OXW6O43L7M6P6FA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.324635396 (499h7m25.195718431s) [noise=true , address=45.91.116.7:4012 , piece_id=DEAO3Z74NO76YYJPBGZABQPISEPXVLRRYGFHN5FCGC5YZJKIM5TA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.521047598 (499h7m25.392130643s) [noise=true , address=77.90.31.10:4520 , piece_id=AFEZB7EPXTQVIY6FWPJ5F4R72Z667KKAMADKVLWBXUHJSJPBHA5A, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.521069500 (499h7m25.392152535s) [seg_pos={0 0}, upload=438074]: Segment reached optimal threshold of 13 pieces. Dec 26 10:36:25.521263278 (499h7m25.392346313s) [noise=true , address=145.79.227.33:4870 , piece_id=2IYNQGXQESUEQDK2CXKX3RTI5S77DQCDT3OGIVYN4UTA2D6N46AQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:25.521273808 (499h7m25.392356853s) [noise=true , address=145.79.227.33:4870 , piece_id=2IYNQGXQESUEQDK2CXKX3RTI5S77DQCDT3OGIVYN4UTA2D6N46AQ, seg_pos={0 0}, upload=438074]: Long tail cancellation due to optimal threshold Dec 26 10:36:25.521281874 (499h7m25.392364919s) [noise=true , address=145.79.227.30:4762 , piece_id=G5HRN3RCGYZW5SXLEWC4PWIX7RALLPWF3Q5EMT7FE3E3NA4B3VEA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:25.521287805 (499h7m25.39237085s) [noise=true , address=145.79.227.30:4762 , piece_id=G5HRN3RCGYZW5SXLEWC4PWIX7RALLPWF3Q5EMT7FE3E3NA4B3VEA, seg_pos={0 0}, upload=438074]: Long tail cancellation due to optimal threshold Dec 26 10:36:25.521488227 (499h7m25.392571272s) [noise=true , address=45.91.116.4:4939 , piece_id=JDLO6ZIHDUJVPVXOICYFTSHZJZGMXWKMZAMB4SOJY6PYGQ2OUKVQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:57376->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:57376->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57376->45.91.116.4:4939: use of closed network connection Dec 26 10:36:25.521494970 (499h7m25.392578015s) [noise=true , address=45.91.116.4:4939 , piece_id=JDLO6ZIHDUJVPVXOICYFTSHZJZGMXWKMZAMB4SOJY6PYGQ2OUKVQ, seg_pos={0 0}, upload=438074]: Long tail cancellation due to optimal threshold Dec 26 10:36:25.521512873 (499h7m25.392595918s) [seg_pos={0 0}, upload=438074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:36:25.521588066 (499h7m25.392671101s) [upload=438074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:25.521593748 (499h7m25.392676783s) [upload=438074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:25.813702260 (499h7m25.684785305s) [upload=438074]: Done uploading segments... Dec 26 10:36:25.813727037 (499h7m25.684810082s) [upload=438074]: Upload finished. err: <nil> Duration: 6.073917282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.........X..................X.......XXX......X............X] duration: 1m0.002852595s trace: https://jaeger.dev.storj.io/trace/689c45a79c742610 FAIL (2025-12-26 10:37:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 10:36:27.577470372 (499h7m27.448553417s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.577672456 (499h7m27.448755491s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578119137 (499h7m27.449202172s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578562491 (499h7m27.449645536s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578686798 (499h7m27.449769843s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.578965638 (499h7m27.450048684s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579314914 (499h7m27.450397959s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579456253 (499h7m27.450539328s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.579866264 (499h7m27.450949299s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580237681 (499h7m27.451320716s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580270013 (499h7m27.451353048s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580573590 (499h7m27.451656635s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.580865527 (499h7m27.451948562s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.581836275 (499h7m27.45291931s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.583318146 (499h7m27.454401191s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:27.585237882 (499h7m27.456320937s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Uploading piece... Dec 26 10:36:28.960965352 (499h7m28.832048387s) [upload=438077]: Next returned nil segment Dec 26 10:36:28.960982154 (499h7m28.832065199s) [upload=438077]: Waiting for error group managing segments... Dec 26 10:36:29.281740992 (499h7m29.152824037s) [noise=true , address=64.49.13.140:4507 , piece_id=62BGQPYPNJLYXIAS5UDYND3YZRKREM7LMWW5D6II4DHOQOMG2IDQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.457737227 (499h7m29.328820262s) [noise=true , address=145.79.227.52:4469 , piece_id=XPTA4L36DLFVSY4NQ4UOFWQADGXAYLUJYWLNMBCLLD6JZM7LJJHQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.468672399 (499h7m29.339755434s) [noise=true , address=145.79.227.43:4189 , piece_id=G3HT7H4TKBCTM6VSCU3XVX3Y5YUJJJHNBW6HQJM2A2TZOQECLLGA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.474132455 (499h7m29.34521549s) [noise=true , address=145.79.227.49:4354 , piece_id=WXFH2GU2GMTNYG2FC3UEORTY2LCBUOYB3WXORXZ4RTSZZJFGIDCQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.563457636 (499h7m29.434540681s) [noise=true , address=64.49.13.170:4718 , piece_id=BVS7VDRLZ7RE2QVIDNTZKHR56OXXEZNX6TI5NLPLOTPLJUYTXKVA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.600957240 (499h7m29.472040275s) [noise=true , address=77.90.31.19:4850 , piece_id=TZ5K25KEAUA2HKMFUK6XBA3UG2SZ3MECBDM6ODH64ALGTRUXPSEQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.633219403 (499h7m29.504302438s) [noise=true , address=77.90.31.11:4575 , piece_id=GBWV6KRQEHZOLOWDQYXQF5C4KZYNH626F65GYBVGROKK3ZITN7JA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.666495797 (499h7m29.537578842s) [noise=true , address=45.91.116.10:4103 , piece_id=UEUFDB5GI7426NRQT7CPEUNXRBOVIYJZMJ7P3KTM7M345FRGUEAQ, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.684666737 (499h7m29.555749772s) [noise=true , address=145.79.227.40:4077 , piece_id=QFB7JU2B5YWKPG7ZNX3VCMG5LYVUXINA7GNNT6U6FLFH4T6UKDEA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.686786163 (499h7m29.557869208s) [noise=true , address=45.91.116.7:4042 , piece_id=4NNXQQGGFYFEKIRHSQDDGZVTHH2SDITBSIXVXZU5KCDQ4BAIYX3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.749653944 (499h7m29.620736989s) [noise=true , address=77.90.31.17:4790 , piece_id=NEM7KTUGPUJUFVO2AY33VYDZH5IVY75ZYSMHORG7SHLVOGNBAC3A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.803202229 (499h7m29.674285274s) [noise=true , address=45.91.116.11:4133 , piece_id=NNMANXPDQVUAXBXVE3YESMWK5SCJ5CZJTOSOCPVP4D3L7HMQDW6Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.812318928 (499h7m29.683401973s) [noise=true , address=45.91.116.9:4272 , piece_id=ZOI6EN53ZTICEXTVBG5J3Y26JWQLK3KJFZIPLJZT4GMIZRQOECNA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884842114 (499h7m29.755925159s) [noise=true , address=77.90.31.18:4828 , piece_id=M66L7JZNAYUIET6GZKJSIU5QIT6LA7D6JGUNAJHDHQDBFF2YJ44A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: <nil> Dec 26 10:36:29.884869236 (499h7m29.755952281s) [seg_pos={0 0}, upload=438077]: Segment reached optimal threshold of 14 pieces. Dec 26 10:36:29.885130354 (499h7m29.756213399s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885139761 (499h7m29.756222806s) [noise=true , address=77.90.31.10:4550 , piece_id=76VLS6BHHLSGPOA5GKNAMXUHMQKF3EGJZDV5IP443S6J45OYRCBA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885215545 (499h7m29.75629859s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885232007 (499h7m29.756315042s) [noise=true , address=145.79.227.30:4777 , piece_id=UTLGEXXCY3BKFT65GCS2GTCUV67SDZQ4NU33T4GWMGKXHXS227HA, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885403694 (499h7m29.756486729s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54232->64.49.13.169:4710: use of closed network connection Dec 26 10:36:29.885410206 (499h7m29.756493242s) [noise=true , address=64.49.13.169:4710 , piece_id=KTOFYC74VOS2PU7S5PBADJ5MU3IDVIYABHP5SFVEFMP3WOIP7J2A, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885477505 (499h7m29.75656055s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:29.885486602 (499h7m29.756569647s) [noise=true , address=77.90.31.13:4653 , piece_id=NGUD3JGWNA7WCVQWOAWBPZVJYCQQUPJZGA3B2YE7TB5T3DQ6W63Q, seg_pos={0 0}, upload=438077]: Long tail cancellation due to optimal threshold Dec 26 10:36:29.885509506 (499h7m29.756592541s) [seg_pos={0 0}, upload=438077]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:36:29.885594277 (499h7m29.756677322s) [upload=438077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:29.885600188 (499h7m29.756683223s) [upload=438077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:30.204831215 (499h7m30.07591425s) [upload=438077]: Done uploading segments... Dec 26 10:36:30.204857135 (499h7m30.07594017s) [upload=438077]: Upload finished. err: <nil> Duration: 1m0.002862193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X....X.] duration: 6.37058285s trace: https://jaeger.dev.storj.io/trace/43c0a796c19ad6f7 FAIL (2025-12-26 10:36:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 10:36:22.815974307 (499h7m22.687057352s) [upload=438074]: Starting upload Dec 26 10:36:22.816042687 (499h7m22.687125732s) [upload=438074]: Done starting upload Dec 26 10:36:22.816185108 (499h7m22.687268153s) [upload=438074]: Uploading segments... Dec 26 10:36:22.816287734 (499h7m22.687370769s) [upload=438074]: Got next segment. Inline: false Dec 26 10:36:22.816333871 (499h7m22.687416916s) [upload=438074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:36:22.816342008 (499h7m22.687425043s) [upload=438074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:36:23.061064477 (499h7m22.932147522s) [seg_pos={0 0}, upload=438074]: Begin upload segment... Dec 26 10:36:23.061116636 (499h7m22.932199681s) [seg_pos={0 0}, upload=438074]: Done begin upload segment. Dec 26 10:36:23.061151703 (499h7m22.932234748s) [noise=true , address=145.79.227.47:4265 , piece_id=BSTBADCISWBMHK7TM7TB5UYECIC5EU3CGVRQWKBWDYWRTIO7KBRA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.061284396 (499h7m22.932367441s) [noise=true , address=145.79.227.33:4870 , piece_id=2IYNQGXQESUEQDK2CXKX3RTI5S77DQCDT3OGIVYN4UTA2D6N46AQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.061943471 (499h7m22.933026506s) [noise=true , address=145.79.227.45:4230 , piece_id=AP3CZ4O4NLI4BHCK6F5QKW7VNA2D3HEL23ZYLTR5OBLMKI7IYHQQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.062468010 (499h7m22.933551055s) [noise=true , address=145.79.227.50:4374 , piece_id=CPNHLDRBRTB2UI4GM2Z6B4ZYO4GCGIUZBLLMFLDQCJC6SVHDB7VQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.062940319 (499h7m22.934023364s) [noise=true , address=45.91.116.4:4939 , piece_id=JDLO6ZIHDUJVPVXOICYFTSHZJZGMXWKMZAMB4SOJY6PYGQ2OUKVQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.063428349 (499h7m22.934511394s) [noise=true , address=145.79.227.48:4302 , piece_id=ZU7VPDLPFK5CPFOG565U5XDNOWDZK6CBLVO4P3YQZTYN3N4BFNPA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.063441884 (499h7m22.934524929s) [noise=true , address=145.79.227.49:4337 , piece_id=LEFMRDLXSVBQJP43BDIHQPM3JHPQJV5GX5L4DU7B4ED3OSES3YPA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.063496488 (499h7m22.934579524s) [noise=true , address=145.79.227.42:4122 , piece_id=C6WD3VRNXRHDISYCSNHGK2JIQXRPKSZYYNXT2MB3MMATNKIYRTTQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.063987173 (499h7m22.935070208s) [noise=true , address=145.79.227.36:4519 , piece_id=WT6JYHVJ2LVDBQGXM45MZAQSZLD5F2ZGHMOV3XKI6WFFH75FNKOQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.064459411 (499h7m22.935542456s) [noise=true , address=145.79.227.6:4259 , piece_id=WM4LUF4D2WGNBAWCKGGVYMSDKPKJGCTU7V3PPWK37GPJD2FXQRTQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.064971787 (499h7m22.936054822s) [noise=true , address=145.79.227.30:4762 , piece_id=G5HRN3RCGYZW5SXLEWC4PWIX7RALLPWF3Q5EMT7FE3E3NA4B3VEA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.065477159 (499h7m22.936560194s) [noise=true , address=145.79.227.43:4159 , piece_id=YVYPM4G7AI7K7AOFC36TDKHRXG3XGDCSJBA6BQRQ5AY72LR4ZZVA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.066065700 (499h7m22.937148745s) [noise=true , address=145.79.227.32:4835 , piece_id=UCKQRSYJQVNVT47KX2I7HP4VQGDIFWDFZIKMWI65TPK34LLET7RQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.066587363 (499h7m22.937670408s) [noise=true , address=77.90.31.10:4520 , piece_id=AFEZB7EPXTQVIY6FWPJ5F4R72Z667KKAMADKVLWBXUHJSJPBHA5A, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.068133508 (499h7m22.939216553s) [noise=true , address=45.91.116.7:4012 , piece_id=DEAO3Z74NO76YYJPBGZABQPISEPXVLRRYGFHN5FCGC5YZJKIM5TA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.068371291 (499h7m22.939454336s) [noise=true , address=77.90.31.11:4556 , piece_id=A7PJWV2ZCBFBGNXDY7XDXL4ZLTMJ4UDZXRKT5OXW6O43L7M6P6FA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:24.734655969 (499h7m24.605739004s) [upload=438074]: Next returned nil segment Dec 26 10:36:24.734670998 (499h7m24.605754033s) [upload=438074]: Waiting for error group managing segments... Dec 26 10:36:25.066064023 (499h7m24.937147059s) [noise=true , address=145.79.227.6:4259 , piece_id=WM4LUF4D2WGNBAWCKGGVYMSDKPKJGCTU7V3PPWK37GPJD2FXQRTQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.070619368 (499h7m24.941702403s) [noise=true , address=145.79.227.47:4265 , piece_id=BSTBADCISWBMHK7TM7TB5UYECIC5EU3CGVRQWKBWDYWRTIO7KBRA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.072315528 (499h7m24.943398573s) [noise=true , address=145.79.227.50:4374 , piece_id=CPNHLDRBRTB2UI4GM2Z6B4ZYO4GCGIUZBLLMFLDQCJC6SVHDB7VQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.072503976 (499h7m24.943587011s) [noise=true , address=145.79.227.49:4337 , piece_id=LEFMRDLXSVBQJP43BDIHQPM3JHPQJV5GX5L4DU7B4ED3OSES3YPA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.073316895 (499h7m24.94439993s) [noise=true , address=145.79.227.43:4159 , piece_id=YVYPM4G7AI7K7AOFC36TDKHRXG3XGDCSJBA6BQRQ5AY72LR4ZZVA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.075446830 (499h7m24.946529876s) [noise=true , address=145.79.227.45:4230 , piece_id=AP3CZ4O4NLI4BHCK6F5QKW7VNA2D3HEL23ZYLTR5OBLMKI7IYHQQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.076975842 (499h7m24.948058887s) [noise=true , address=145.79.227.48:4302 , piece_id=ZU7VPDLPFK5CPFOG565U5XDNOWDZK6CBLVO4P3YQZTYN3N4BFNPA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.088131574 (499h7m24.959214619s) [noise=true , address=145.79.227.42:4122 , piece_id=C6WD3VRNXRHDISYCSNHGK2JIQXRPKSZYYNXT2MB3MMATNKIYRTTQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.099031991 (499h7m24.970115036s) [noise=true , address=145.79.227.32:4835 , piece_id=UCKQRSYJQVNVT47KX2I7HP4VQGDIFWDFZIKMWI65TPK34LLET7RQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.100283463 (499h7m24.971366498s) [noise=true , address=145.79.227.36:4519 , piece_id=WT6JYHVJ2LVDBQGXM45MZAQSZLD5F2ZGHMOV3XKI6WFFH75FNKOQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.317502444 (499h7m25.188585489s) [noise=true , address=77.90.31.11:4556 , piece_id=A7PJWV2ZCBFBGNXDY7XDXL4ZLTMJ4UDZXRKT5OXW6O43L7M6P6FA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.324635396 (499h7m25.195718431s) [noise=true , address=45.91.116.7:4012 , piece_id=DEAO3Z74NO76YYJPBGZABQPISEPXVLRRYGFHN5FCGC5YZJKIM5TA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.521047598 (499h7m25.392130643s) [noise=true , address=77.90.31.10:4520 , piece_id=AFEZB7EPXTQVIY6FWPJ5F4R72Z667KKAMADKVLWBXUHJSJPBHA5A, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.521069500 (499h7m25.392152535s) [seg_pos={0 0}, upload=438074]: Segment reached optimal threshold of 13 pieces. Dec 26 10:36:25.521263278 (499h7m25.392346313s) [noise=true , address=145.79.227.33:4870 , piece_id=2IYNQGXQESUEQDK2CXKX3RTI5S77DQCDT3OGIVYN4UTA2D6N46AQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:25.521273808 (499h7m25.392356853s) [noise=true , address=145.79.227.33:4870 , piece_id=2IYNQGXQESUEQDK2CXKX3RTI5S77DQCDT3OGIVYN4UTA2D6N46AQ, seg_pos={0 0}, upload=438074]: Long tail cancellation due to optimal threshold Dec 26 10:36:25.521281874 (499h7m25.392364919s) [noise=true , address=145.79.227.30:4762 , piece_id=G5HRN3RCGYZW5SXLEWC4PWIX7RALLPWF3Q5EMT7FE3E3NA4B3VEA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:25.521287805 (499h7m25.39237085s) [noise=true , address=145.79.227.30:4762 , piece_id=G5HRN3RCGYZW5SXLEWC4PWIX7RALLPWF3Q5EMT7FE3E3NA4B3VEA, seg_pos={0 0}, upload=438074]: Long tail cancellation due to optimal threshold Dec 26 10:36:25.521488227 (499h7m25.392571272s) [noise=true , address=45.91.116.4:4939 , piece_id=JDLO6ZIHDUJVPVXOICYFTSHZJZGMXWKMZAMB4SOJY6PYGQ2OUKVQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:57376->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:57376->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57376->45.91.116.4:4939: use of closed network connection Dec 26 10:36:25.521494970 (499h7m25.392578015s) [noise=true , address=45.91.116.4:4939 , piece_id=JDLO6ZIHDUJVPVXOICYFTSHZJZGMXWKMZAMB4SOJY6PYGQ2OUKVQ, seg_pos={0 0}, upload=438074]: Long tail cancellation due to optimal threshold Dec 26 10:36:25.521512873 (499h7m25.392595918s) [seg_pos={0 0}, upload=438074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:36:25.521588066 (499h7m25.392671101s) [upload=438074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:25.521593748 (499h7m25.392676783s) [upload=438074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:25.813702260 (499h7m25.684785305s) [upload=438074]: Done uploading segments... Dec 26 10:36:25.813727037 (499h7m25.684810082s) [upload=438074]: Upload finished. err: <nil> Duration: 6.370600253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X....X] duration: 30.00338507s trace: https://jaeger.dev.storj.io/trace/43c0a796c19ad6f7 FAIL (2025-12-26 10:36:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 10:36:22.815974307 (499h7m22.687057352s) [upload=438074]: Starting upload Dec 26 10:36:22.816042687 (499h7m22.687125732s) [upload=438074]: Done starting upload Dec 26 10:36:22.816185108 (499h7m22.687268153s) [upload=438074]: Uploading segments... Dec 26 10:36:22.816287734 (499h7m22.687370769s) [upload=438074]: Got next segment. Inline: false Dec 26 10:36:22.816333871 (499h7m22.687416916s) [upload=438074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:36:22.816342008 (499h7m22.687425043s) [upload=438074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:36:23.061064477 (499h7m22.932147522s) [seg_pos={0 0}, upload=438074]: Begin upload segment... Dec 26 10:36:23.061116636 (499h7m22.932199681s) [seg_pos={0 0}, upload=438074]: Done begin upload segment. Dec 26 10:36:23.061151703 (499h7m22.932234748s) [noise=true , address=145.79.227.47:4265 , piece_id=BSTBADCISWBMHK7TM7TB5UYECIC5EU3CGVRQWKBWDYWRTIO7KBRA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.061284396 (499h7m22.932367441s) [noise=true , address=145.79.227.33:4870 , piece_id=2IYNQGXQESUEQDK2CXKX3RTI5S77DQCDT3OGIVYN4UTA2D6N46AQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.061943471 (499h7m22.933026506s) [noise=true , address=145.79.227.45:4230 , piece_id=AP3CZ4O4NLI4BHCK6F5QKW7VNA2D3HEL23ZYLTR5OBLMKI7IYHQQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.062468010 (499h7m22.933551055s) [noise=true , address=145.79.227.50:4374 , piece_id=CPNHLDRBRTB2UI4GM2Z6B4ZYO4GCGIUZBLLMFLDQCJC6SVHDB7VQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.062940319 (499h7m22.934023364s) [noise=true , address=45.91.116.4:4939 , piece_id=JDLO6ZIHDUJVPVXOICYFTSHZJZGMXWKMZAMB4SOJY6PYGQ2OUKVQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.063428349 (499h7m22.934511394s) [noise=true , address=145.79.227.48:4302 , piece_id=ZU7VPDLPFK5CPFOG565U5XDNOWDZK6CBLVO4P3YQZTYN3N4BFNPA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.063441884 (499h7m22.934524929s) [noise=true , address=145.79.227.49:4337 , piece_id=LEFMRDLXSVBQJP43BDIHQPM3JHPQJV5GX5L4DU7B4ED3OSES3YPA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.063496488 (499h7m22.934579524s) [noise=true , address=145.79.227.42:4122 , piece_id=C6WD3VRNXRHDISYCSNHGK2JIQXRPKSZYYNXT2MB3MMATNKIYRTTQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.063987173 (499h7m22.935070208s) [noise=true , address=145.79.227.36:4519 , piece_id=WT6JYHVJ2LVDBQGXM45MZAQSZLD5F2ZGHMOV3XKI6WFFH75FNKOQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.064459411 (499h7m22.935542456s) [noise=true , address=145.79.227.6:4259 , piece_id=WM4LUF4D2WGNBAWCKGGVYMSDKPKJGCTU7V3PPWK37GPJD2FXQRTQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.064971787 (499h7m22.936054822s) [noise=true , address=145.79.227.30:4762 , piece_id=G5HRN3RCGYZW5SXLEWC4PWIX7RALLPWF3Q5EMT7FE3E3NA4B3VEA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.065477159 (499h7m22.936560194s) [noise=true , address=145.79.227.43:4159 , piece_id=YVYPM4G7AI7K7AOFC36TDKHRXG3XGDCSJBA6BQRQ5AY72LR4ZZVA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.066065700 (499h7m22.937148745s) [noise=true , address=145.79.227.32:4835 , piece_id=UCKQRSYJQVNVT47KX2I7HP4VQGDIFWDFZIKMWI65TPK34LLET7RQ, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.066587363 (499h7m22.937670408s) [noise=true , address=77.90.31.10:4520 , piece_id=AFEZB7EPXTQVIY6FWPJ5F4R72Z667KKAMADKVLWBXUHJSJPBHA5A, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.068133508 (499h7m22.939216553s) [noise=true , address=45.91.116.7:4012 , piece_id=DEAO3Z74NO76YYJPBGZABQPISEPXVLRRYGFHN5FCGC5YZJKIM5TA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:23.068371291 (499h7m22.939454336s) [noise=true , address=77.90.31.11:4556 , piece_id=A7PJWV2ZCBFBGNXDY7XDXL4ZLTMJ4UDZXRKT5OXW6O43L7M6P6FA, seg_pos={0 0}, upload=438074]: Uploading piece... Dec 26 10:36:24.734655969 (499h7m24.605739004s) [upload=438074]: Next returned nil segment Dec 26 10:36:24.734670998 (499h7m24.605754033s) [upload=438074]: Waiting for error group managing segments... Dec 26 10:36:25.066064023 (499h7m24.937147059s) [noise=true , address=145.79.227.6:4259 , piece_id=WM4LUF4D2WGNBAWCKGGVYMSDKPKJGCTU7V3PPWK37GPJD2FXQRTQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.070619368 (499h7m24.941702403s) [noise=true , address=145.79.227.47:4265 , piece_id=BSTBADCISWBMHK7TM7TB5UYECIC5EU3CGVRQWKBWDYWRTIO7KBRA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.072315528 (499h7m24.943398573s) [noise=true , address=145.79.227.50:4374 , piece_id=CPNHLDRBRTB2UI4GM2Z6B4ZYO4GCGIUZBLLMFLDQCJC6SVHDB7VQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.072503976 (499h7m24.943587011s) [noise=true , address=145.79.227.49:4337 , piece_id=LEFMRDLXSVBQJP43BDIHQPM3JHPQJV5GX5L4DU7B4ED3OSES3YPA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.073316895 (499h7m24.94439993s) [noise=true , address=145.79.227.43:4159 , piece_id=YVYPM4G7AI7K7AOFC36TDKHRXG3XGDCSJBA6BQRQ5AY72LR4ZZVA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.075446830 (499h7m24.946529876s) [noise=true , address=145.79.227.45:4230 , piece_id=AP3CZ4O4NLI4BHCK6F5QKW7VNA2D3HEL23ZYLTR5OBLMKI7IYHQQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.076975842 (499h7m24.948058887s) [noise=true , address=145.79.227.48:4302 , piece_id=ZU7VPDLPFK5CPFOG565U5XDNOWDZK6CBLVO4P3YQZTYN3N4BFNPA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.088131574 (499h7m24.959214619s) [noise=true , address=145.79.227.42:4122 , piece_id=C6WD3VRNXRHDISYCSNHGK2JIQXRPKSZYYNXT2MB3MMATNKIYRTTQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.099031991 (499h7m24.970115036s) [noise=true , address=145.79.227.32:4835 , piece_id=UCKQRSYJQVNVT47KX2I7HP4VQGDIFWDFZIKMWI65TPK34LLET7RQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.100283463 (499h7m24.971366498s) [noise=true , address=145.79.227.36:4519 , piece_id=WT6JYHVJ2LVDBQGXM45MZAQSZLD5F2ZGHMOV3XKI6WFFH75FNKOQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.317502444 (499h7m25.188585489s) [noise=true , address=77.90.31.11:4556 , piece_id=A7PJWV2ZCBFBGNXDY7XDXL4ZLTMJ4UDZXRKT5OXW6O43L7M6P6FA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.324635396 (499h7m25.195718431s) [noise=true , address=45.91.116.7:4012 , piece_id=DEAO3Z74NO76YYJPBGZABQPISEPXVLRRYGFHN5FCGC5YZJKIM5TA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.521047598 (499h7m25.392130643s) [noise=true , address=77.90.31.10:4520 , piece_id=AFEZB7EPXTQVIY6FWPJ5F4R72Z667KKAMADKVLWBXUHJSJPBHA5A, seg_pos={0 0}, upload=438074]: Done uploading piece. err: <nil> Dec 26 10:36:25.521069500 (499h7m25.392152535s) [seg_pos={0 0}, upload=438074]: Segment reached optimal threshold of 13 pieces. Dec 26 10:36:25.521263278 (499h7m25.392346313s) [noise=true , address=145.79.227.33:4870 , piece_id=2IYNQGXQESUEQDK2CXKX3RTI5S77DQCDT3OGIVYN4UTA2D6N46AQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:25.521273808 (499h7m25.392356853s) [noise=true , address=145.79.227.33:4870 , piece_id=2IYNQGXQESUEQDK2CXKX3RTI5S77DQCDT3OGIVYN4UTA2D6N46AQ, seg_pos={0 0}, upload=438074]: Long tail cancellation due to optimal threshold Dec 26 10:36:25.521281874 (499h7m25.392364919s) [noise=true , address=145.79.227.30:4762 , piece_id=G5HRN3RCGYZW5SXLEWC4PWIX7RALLPWF3Q5EMT7FE3E3NA4B3VEA, seg_pos={0 0}, upload=438074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 10:36:25.521287805 (499h7m25.39237085s) [noise=true , address=145.79.227.30:4762 , piece_id=G5HRN3RCGYZW5SXLEWC4PWIX7RALLPWF3Q5EMT7FE3E3NA4B3VEA, seg_pos={0 0}, upload=438074]: Long tail cancellation due to optimal threshold Dec 26 10:36:25.521488227 (499h7m25.392571272s) [noise=true , address=45.91.116.4:4939 , piece_id=JDLO6ZIHDUJVPVXOICYFTSHZJZGMXWKMZAMB4SOJY6PYGQ2OUKVQ, seg_pos={0 0}, upload=438074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:57376->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:57376->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57376->45.91.116.4:4939: use of closed network connection Dec 26 10:36:25.521494970 (499h7m25.392578015s) [noise=true , address=45.91.116.4:4939 , piece_id=JDLO6ZIHDUJVPVXOICYFTSHZJZGMXWKMZAMB4SOJY6PYGQ2OUKVQ, seg_pos={0 0}, upload=438074]: Long tail cancellation due to optimal threshold Dec 26 10:36:25.521512873 (499h7m25.392595918s) [seg_pos={0 0}, upload=438074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:36:25.521588066 (499h7m25.392671101s) [upload=438074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:36:25.521593748 (499h7m25.392676783s) [upload=438074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:36:25.813702260 (499h7m25.684785305s) [upload=438074]: Done uploading segments... Dec 26 10:36:25.813727037 (499h7m25.684810082s) [upload=438074]: Upload finished. err: <nil> Duration: 30.003408223s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.........X..................X.......XXX......X............] duration: 46.961625419s trace: https://jaeger.dev.storj.io/trace/7e25338f73046b2a FAIL (2025-12-26 10:36:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033130872 (498h55m22.904213918s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033471341 (498h55m22.904554386s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033892292 (498h55m22.904975328s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034387946 (498h55m22.905470991s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034807164 (498h55m22.905890209s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035259836 (498h55m22.906342881s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035727828 (498h55m22.906810863s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036033871 (498h55m22.907116906s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036579810 (498h55m22.907662855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.037767882 (498h55m22.908850927s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.038064076 (498h55m22.909147122s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.039224646 (498h55m22.910307691s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040040249 (498h55m22.911123294s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040366420 (498h55m22.911449455s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.041569130 (498h55m22.912652175s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:24.513239450 (498h55m24.384322485s) [upload=437904]: Next returned nil segment Dec 26 10:24:24.513254640 (498h55m24.384337685s) [upload=437904]: Waiting for error group managing segments... Dec 26 10:24:24.926029601 (498h55m24.797112636s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.027458408 (498h55m24.898541443s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.094888666 (498h55m24.965971701s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.102988388 (498h55m24.974071433s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.109420056 (498h55m24.980503091s) [noise=true , address=77.90.31.15:4732 , piece_id=QDQXC2LEVGS7K5OHDX7Q3ZETJJ37XFXIS6BXXNFCUTKFEMDLJ23A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.122147140 (498h55m24.993230186s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.187517306 (498h55m25.058600351s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.195537778 (498h55m25.066620823s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.255475810 (498h55m25.126558855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.546862739 (498h55m25.417945784s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.792060605 (498h55m25.66314365s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.809945911 (498h55m25.681028946s) [noise=true , address=145.79.227.44:4221 , piece_id=IZLW4HA64OMYW23ZBPBZU45GR4QCEREPUL76QL6RWWUA3LKKFN2A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.860843739 (498h55m25.731926774s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861204966 (498h55m25.732288001s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861232289 (498h55m25.732315324s) [seg_pos={0 0}, upload=437904]: Segment reached optimal threshold of 14 pieces. Dec 26 10:24:25.861471414 (498h55m25.732554449s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861479399 (498h55m25.732562434s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861580682 (498h55m25.732663717s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection Dec 26 10:24:25.861589008 (498h55m25.732672043s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861840307 (498h55m25.732923342s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861847050 (498h55m25.732930085s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861929887 (498h55m25.733012932s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861939285 (498h55m25.73302232s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861959854 (498h55m25.733042889s) [seg_pos={0 0}, upload=437904]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:24:25.862033124 (498h55m25.733116159s) [upload=437904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:24:25.862037923 (498h55m25.733120958s) [upload=437904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:24:26.151364428 (498h55m26.022447463s) [upload=437904]: Done uploading segments... Dec 26 10:24:26.151393914 (498h55m26.022476949s) [upload=437904]: Upload finished. err: <nil> Duration: 46.961646841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X....] duration: 5.714869412s trace: https://jaeger.dev.storj.io/trace/6043e12ffff903ef FAIL (2025-12-26 10:36:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =438015]: Starting upload Dec 26 10:32:27.503304645 (499h3m27.37438768s) [upload=438015]: Done starting upload Dec 26 10:32:27.503555723 (499h3m27.374638768s) [upload=438015]: Uploading segments... Dec 26 10:32:27.503599195 (499h3m27.37468223s) [upload=438015]: Got next segment. Inline: false Dec 26 10:32:27.503644833 (499h3m27.374727868s) [upload=438015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:32:27.503648870 (499h3m27.374731905s) [upload=438015]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:32:27.745476991 (499h3m27.616560036s) [seg_pos={0 0}, upload=438015]: Begin upload segment... Dec 26 10:32:27.745536694 (499h3m27.616619739s) [seg_pos={0 0}, upload=438015]: Done begin upload segment. Dec 26 10:32:27.745564968 (499h3m27.616648003s) [noise=true , address=145.79.227.39:4014 , piece_id=XOZYZ6IW7VTHODKCE47BBH6TRCJBNZAIOQECIA7BYN6OF7MXKAOA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.745572803 (499h3m27.616655838s) [noise=true , address=145.79.227.15:4876 , piece_id=MC5RYVNACVGCQLJ3D3HK3QGNRZ6WMOYWEP2ELPJIAM2UNBZTZYXA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.746267105 (499h3m27.61735015s) [noise=true , address=45.91.116.6:4975 , piece_id=ZXWQMTVELBYUB2FAYFDI2OEPHACFEZK5T3DXNZXFFH4RX7CL5EEQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.746839796 (499h3m27.617922841s) [noise=true , address=145.79.227.38:4977 , piece_id=MNIKKIRFDNGHMK4YANINDCZSBRWOYSMBBD6375JGPYY3NBYSF3VA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.747186877 (499h3m27.618269912s) [noise=true , address=145.79.227.52:4446 , piece_id=QM5DJJEGJ4U44YTHMS4DUPJUOYX6TO7H24P2LE3EMNVEMEYLRM4A, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.747573212 (499h3m27.618656257s) [noise=true , address=145.79.227.6:4258 , piece_id=DN6GTGMTXK3I2XBC7AW425E4RCFXMJ37HYIFCNW7LLPIVRYJCKVQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.747937996 (499h3m27.619021031s) [noise=true , address=45.91.116.7:4010 , piece_id=S6LZG5MC6A6NURSD5N2OF2X74DMCG5GN4CKB2ZPSVBSQGF6MUG4A, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.748273225 (499h3m27.61935627s) [noise=true , address=145.79.227.13:4802 , piece_id=GKZKQCAD7PXCTOADWCX4ADN5JFFYQPGODHWN3WQGFC7HZKHUXL5A, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.749530719 (499h3m27.620613765s) [noise=true , address=145.79.227.29:4942 , piece_id=DPXUJQJ3BJZPYR3GXLT6GH3DUMUKOZQIBTUQHTKANNVKTG6RLUPQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.749561648 (499h3m27.620644694s) [noise=true , address=145.79.227.46:4603 , piece_id=5XTSPOFXJ5WG77MVXSQVY53GRJ3M3CCD3GW3PT22SOX6QCJFDNHQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.751321760 (499h3m27.622404806s) [noise=true , address=145.79.227.45:4230 , piece_id=27H4IQBNLB4DTQGH3MDE3P6WBBH7GKO33I3ZDWY25UGFDFPDNBXQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.753391351 (499h3m27.624474386s) [noise=true , address=77.90.31.17:4770 , piece_id=NY52IIK2M7HQSRCYFLJROHKUZ5U5FPCPXGQGURWPPEJZASEPJYUQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.755285377 (499h3m27.626368422s) [noise=true , address=45.91.116.11:4120 , piece_id=3LTTNZIS3YPJ6OBKE3Q76ETFDVI2SUPESHBCLFVUH2CGUTLEZYCQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.757601458 (499h3m27.628684503s) [noise=true , address=77.90.31.15:4700 , piece_id=O72QP657I6TF6A7U6OQLFA7YON4C2FSSJZP4NQYMHD464QNXQUMQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.759623167 (499h3m27.630706202s) [noise=true , address=145.79.227.4:4185 , piece_id=JOLBEJZMIC5UTTOGNS654EZVIYANB5UE6NLGJKOVP2JGJ3QYWCFQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.760142336 (499h3m27.631225371s) [noise=true , address=145.79.227.48:4301 , piece_id=G46SWJPXKAMMN6JZTBYK27TUBY7W4LUOXASJK36LQS6MAXF6E5XA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:29.467859286 (499h3m29.338942341s) [upload=438015]: Next returned nil segment Dec 26 10:32:29.467876819 (499h3m29.338959864s) [upload=438015]: Waiting for error group managing segments... Dec 26 10:32:29.756063633 (499h3m29.627146668s) [noise=true , address=145.79.227.38:4977 , piece_id=MNIKKIRFDNGHMK4YANINDCZSBRWOYSMBBD6375JGPYY3NBYSF3VA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.756671280 (499h3m29.627754325s) [noise=true , address=145.79.227.15:4876 , piece_id=MC5RYVNACVGCQLJ3D3HK3QGNRZ6WMOYWEP2ELPJIAM2UNBZTZYXA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.757161754 (499h3m29.628244799s) [noise=true , address=145.79.227.29:4942 , piece_id=DPXUJQJ3BJZPYR3GXLT6GH3DUMUKOZQIBTUQHTKANNVKTG6RLUPQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.757778659 (499h3m29.628861704s) [noise=true , address=145.79.227.13:4802 , piece_id=GKZKQCAD7PXCTOADWCX4ADN5JFFYQPGODHWN3WQGFC7HZKHUXL5A, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.757996383 (499h3m29.629079428s) [noise=true , address=145.79.227.52:4446 , piece_id=QM5DJJEGJ4U44YTHMS4DUPJUOYX6TO7H24P2LE3EMNVEMEYLRM4A, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.763864007 (499h3m29.634947052s) [noise=true , address=145.79.227.6:4258 , piece_id=DN6GTGMTXK3I2XBC7AW425E4RCFXMJ37HYIFCNW7LLPIVRYJCKVQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.764489287 (499h3m29.635572332s) [noise=true , address=145.79.227.48:4301 , piece_id=G46SWJPXKAMMN6JZTBYK27TUBY7W4LUOXASJK36LQS6MAXF6E5XA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.764947540 (499h3m29.636030585s) [noise=true , address=145.79.227.39:4014 , piece_id=XOZYZ6IW7VTHODKCE47BBH6TRCJBNZAIOQECIA7BYN6OF7MXKAOA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.780700537 (499h3m29.651783582s) [noise=true , address=145.79.227.4:4185 , piece_id=JOLBEJZMIC5UTTOGNS654EZVIYANB5UE6NLGJKOVP2JGJ3QYWCFQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.785974919 (499h3m29.657057964s) [noise=true , address=145.79.227.45:4230 , piece_id=27H4IQBNLB4DTQGH3MDE3P6WBBH7GKO33I3ZDWY25UGFDFPDNBXQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.791776337 (499h3m29.662859382s) [noise=true , address=145.79.227.46:4603 , piece_id=5XTSPOFXJ5WG77MVXSQVY53GRJ3M3CCD3GW3PT22SOX6QCJFDNHQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:30.037534750 (499h3m29.908617785s) [noise=true , address=45.91.116.7:4010 , piece_id=S6LZG5MC6A6NURSD5N2OF2X74DMCG5GN4CKB2ZPSVBSQGF6MUG4A, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:30.061182980 (499h3m29.932266015s) [noise=true , address=45.91.116.6:4975 , piece_id=ZXWQMTVELBYUB2FAYFDI2OEPHACFEZK5T3DXNZXFFH4RX7CL5EEQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:30.061271860 (499h3m29.932354895s) [seg_pos={0 0}, upload=438015]: Segment reached optimal threshold of 13 pieces. Dec 26 10:32:30.061580838 (499h3m29.932663883s) [noise=true , address=45.91.116.11:4120 , piece_id=3LTTNZIS3YPJ6OBKE3Q76ETFDVI2SUPESHBCLFVUH2CGUTLEZYCQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 10:32:30.061620123 (499h3m29.932703188s) [noise=true , address=45.91.116.11:4120 , piece_id=3LTTNZIS3YPJ6OBKE3Q76ETFDVI2SUPESHBCLFVUH2CGUTLEZYCQ, seg_pos={0 0}, upload=438015]: Long tail cancellation due to optimal threshold Dec 26 10:32:30.061682210 (499h3m29.932765256s) [noise=true , address=77.90.31.15:4700 , piece_id=O72QP657I6TF6A7U6OQLFA7YON4C2FSSJZP4NQYMHD464QNXQUMQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: read tcp 188.166.225.170:41482->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:32:30.061705164 (499h3m29.9327882s) [noise=true , address=77.90.31.15:4700 , piece_id=O72QP657I6TF6A7U6OQLFA7YON4C2FSSJZP4NQYMHD464QNXQUMQ, seg_pos={0 0}, upload=438015]: Long tail cancellation due to optimal threshold Dec 26 10:32:30.061763296 (499h3m29.932846331s) [noise=true , address=77.90.31.17:4770 , piece_id=NY52IIK2M7HQSRCYFLJROHKUZ5U5FPCPXGQGURWPPEJZASEPJYUQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:45072->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:45072->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45072->77.90.31.17:4770: use of closed network connection Dec 26 10:32:30.061770028 (499h3m29.932853063s) [noise=true , address=77.90.31.17:4770 , piece_id=NY52IIK2M7HQSRCYFLJROHKUZ5U5FPCPXGQGURWPPEJZASEPJYUQ, seg_pos={0 0}, upload=438015]: Long tail cancellation due to optimal threshold Dec 26 10:32:30.061785908 (499h3m29.932868953s) [seg_pos={0 0}, upload=438015]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:32:30.061895307 (499h3m29.932978352s) [upload=438015]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:32:30.061899966 (499h3m29.932983001s) [upload=438015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:32:30.345947669 (499h3m30.217030714s) [upload=438015]: Done uploading segments... Dec 26 10:32:30.345971524 (499h3m30.217054559s) [upload=438015]: Upload finished. err: <nil> Duration: 5.714896504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.........X..................X.......XXX......X...........] duration: 56.027659248s trace: https://jaeger.dev.storj.io/trace/7e25338f73046b2a FAIL (2025-12-26 10:35:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033130872 (498h55m22.904213918s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033471341 (498h55m22.904554386s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033892292 (498h55m22.904975328s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034387946 (498h55m22.905470991s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034807164 (498h55m22.905890209s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035259836 (498h55m22.906342881s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035727828 (498h55m22.906810863s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036033871 (498h55m22.907116906s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036579810 (498h55m22.907662855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.037767882 (498h55m22.908850927s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.038064076 (498h55m22.909147122s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.039224646 (498h55m22.910307691s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040040249 (498h55m22.911123294s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040366420 (498h55m22.911449455s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.041569130 (498h55m22.912652175s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:24.513239450 (498h55m24.384322485s) [upload=437904]: Next returned nil segment Dec 26 10:24:24.513254640 (498h55m24.384337685s) [upload=437904]: Waiting for error group managing segments... Dec 26 10:24:24.926029601 (498h55m24.797112636s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.027458408 (498h55m24.898541443s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.094888666 (498h55m24.965971701s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.102988388 (498h55m24.974071433s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.109420056 (498h55m24.980503091s) [noise=true , address=77.90.31.15:4732 , piece_id=QDQXC2LEVGS7K5OHDX7Q3ZETJJ37XFXIS6BXXNFCUTKFEMDLJ23A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.122147140 (498h55m24.993230186s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.187517306 (498h55m25.058600351s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.195537778 (498h55m25.066620823s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.255475810 (498h55m25.126558855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.546862739 (498h55m25.417945784s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.792060605 (498h55m25.66314365s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.809945911 (498h55m25.681028946s) [noise=true , address=145.79.227.44:4221 , piece_id=IZLW4HA64OMYW23ZBPBZU45GR4QCEREPUL76QL6RWWUA3LKKFN2A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.860843739 (498h55m25.731926774s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861204966 (498h55m25.732288001s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861232289 (498h55m25.732315324s) [seg_pos={0 0}, upload=437904]: Segment reached optimal threshold of 14 pieces. Dec 26 10:24:25.861471414 (498h55m25.732554449s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861479399 (498h55m25.732562434s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861580682 (498h55m25.732663717s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection Dec 26 10:24:25.861589008 (498h55m25.732672043s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861840307 (498h55m25.732923342s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861847050 (498h55m25.732930085s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861929887 (498h55m25.733012932s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861939285 (498h55m25.73302232s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861959854 (498h55m25.733042889s) [seg_pos={0 0}, upload=437904]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:24:25.862033124 (498h55m25.733116159s) [upload=437904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:24:25.862037923 (498h55m25.733120958s) [upload=437904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:24:26.151364428 (498h55m26.022447463s) [upload=437904]: Done uploading segments... Dec 26 10:24:26.151393914 (498h55m26.022476949s) [upload=437904]: Upload finished. err: <nil> Duration: 56.027710596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X...] duration: 5.673426558s trace: https://jaeger.dev.storj.io/trace/6043e12ffff903ef FAIL (2025-12-26 10:35:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =438015]: Starting upload Dec 26 10:32:27.503304645 (499h3m27.37438768s) [upload=438015]: Done starting upload Dec 26 10:32:27.503555723 (499h3m27.374638768s) [upload=438015]: Uploading segments... Dec 26 10:32:27.503599195 (499h3m27.37468223s) [upload=438015]: Got next segment. Inline: false Dec 26 10:32:27.503644833 (499h3m27.374727868s) [upload=438015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:32:27.503648870 (499h3m27.374731905s) [upload=438015]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:32:27.745476991 (499h3m27.616560036s) [seg_pos={0 0}, upload=438015]: Begin upload segment... Dec 26 10:32:27.745536694 (499h3m27.616619739s) [seg_pos={0 0}, upload=438015]: Done begin upload segment. Dec 26 10:32:27.745564968 (499h3m27.616648003s) [noise=true , address=145.79.227.39:4014 , piece_id=XOZYZ6IW7VTHODKCE47BBH6TRCJBNZAIOQECIA7BYN6OF7MXKAOA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.745572803 (499h3m27.616655838s) [noise=true , address=145.79.227.15:4876 , piece_id=MC5RYVNACVGCQLJ3D3HK3QGNRZ6WMOYWEP2ELPJIAM2UNBZTZYXA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.746267105 (499h3m27.61735015s) [noise=true , address=45.91.116.6:4975 , piece_id=ZXWQMTVELBYUB2FAYFDI2OEPHACFEZK5T3DXNZXFFH4RX7CL5EEQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.746839796 (499h3m27.617922841s) [noise=true , address=145.79.227.38:4977 , piece_id=MNIKKIRFDNGHMK4YANINDCZSBRWOYSMBBD6375JGPYY3NBYSF3VA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.747186877 (499h3m27.618269912s) [noise=true , address=145.79.227.52:4446 , piece_id=QM5DJJEGJ4U44YTHMS4DUPJUOYX6TO7H24P2LE3EMNVEMEYLRM4A, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.747573212 (499h3m27.618656257s) [noise=true , address=145.79.227.6:4258 , piece_id=DN6GTGMTXK3I2XBC7AW425E4RCFXMJ37HYIFCNW7LLPIVRYJCKVQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.747937996 (499h3m27.619021031s) [noise=true , address=45.91.116.7:4010 , piece_id=S6LZG5MC6A6NURSD5N2OF2X74DMCG5GN4CKB2ZPSVBSQGF6MUG4A, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.748273225 (499h3m27.61935627s) [noise=true , address=145.79.227.13:4802 , piece_id=GKZKQCAD7PXCTOADWCX4ADN5JFFYQPGODHWN3WQGFC7HZKHUXL5A, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.749530719 (499h3m27.620613765s) [noise=true , address=145.79.227.29:4942 , piece_id=DPXUJQJ3BJZPYR3GXLT6GH3DUMUKOZQIBTUQHTKANNVKTG6RLUPQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.749561648 (499h3m27.620644694s) [noise=true , address=145.79.227.46:4603 , piece_id=5XTSPOFXJ5WG77MVXSQVY53GRJ3M3CCD3GW3PT22SOX6QCJFDNHQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.751321760 (499h3m27.622404806s) [noise=true , address=145.79.227.45:4230 , piece_id=27H4IQBNLB4DTQGH3MDE3P6WBBH7GKO33I3ZDWY25UGFDFPDNBXQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.753391351 (499h3m27.624474386s) [noise=true , address=77.90.31.17:4770 , piece_id=NY52IIK2M7HQSRCYFLJROHKUZ5U5FPCPXGQGURWPPEJZASEPJYUQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.755285377 (499h3m27.626368422s) [noise=true , address=45.91.116.11:4120 , piece_id=3LTTNZIS3YPJ6OBKE3Q76ETFDVI2SUPESHBCLFVUH2CGUTLEZYCQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.757601458 (499h3m27.628684503s) [noise=true , address=77.90.31.15:4700 , piece_id=O72QP657I6TF6A7U6OQLFA7YON4C2FSSJZP4NQYMHD464QNXQUMQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.759623167 (499h3m27.630706202s) [noise=true , address=145.79.227.4:4185 , piece_id=JOLBEJZMIC5UTTOGNS654EZVIYANB5UE6NLGJKOVP2JGJ3QYWCFQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.760142336 (499h3m27.631225371s) [noise=true , address=145.79.227.48:4301 , piece_id=G46SWJPXKAMMN6JZTBYK27TUBY7W4LUOXASJK36LQS6MAXF6E5XA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:29.467859286 (499h3m29.338942341s) [upload=438015]: Next returned nil segment Dec 26 10:32:29.467876819 (499h3m29.338959864s) [upload=438015]: Waiting for error group managing segments... Dec 26 10:32:29.756063633 (499h3m29.627146668s) [noise=true , address=145.79.227.38:4977 , piece_id=MNIKKIRFDNGHMK4YANINDCZSBRWOYSMBBD6375JGPYY3NBYSF3VA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.756671280 (499h3m29.627754325s) [noise=true , address=145.79.227.15:4876 , piece_id=MC5RYVNACVGCQLJ3D3HK3QGNRZ6WMOYWEP2ELPJIAM2UNBZTZYXA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.757161754 (499h3m29.628244799s) [noise=true , address=145.79.227.29:4942 , piece_id=DPXUJQJ3BJZPYR3GXLT6GH3DUMUKOZQIBTUQHTKANNVKTG6RLUPQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.757778659 (499h3m29.628861704s) [noise=true , address=145.79.227.13:4802 , piece_id=GKZKQCAD7PXCTOADWCX4ADN5JFFYQPGODHWN3WQGFC7HZKHUXL5A, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.757996383 (499h3m29.629079428s) [noise=true , address=145.79.227.52:4446 , piece_id=QM5DJJEGJ4U44YTHMS4DUPJUOYX6TO7H24P2LE3EMNVEMEYLRM4A, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.763864007 (499h3m29.634947052s) [noise=true , address=145.79.227.6:4258 , piece_id=DN6GTGMTXK3I2XBC7AW425E4RCFXMJ37HYIFCNW7LLPIVRYJCKVQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.764489287 (499h3m29.635572332s) [noise=true , address=145.79.227.48:4301 , piece_id=G46SWJPXKAMMN6JZTBYK27TUBY7W4LUOXASJK36LQS6MAXF6E5XA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.764947540 (499h3m29.636030585s) [noise=true , address=145.79.227.39:4014 , piece_id=XOZYZ6IW7VTHODKCE47BBH6TRCJBNZAIOQECIA7BYN6OF7MXKAOA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.780700537 (499h3m29.651783582s) [noise=true , address=145.79.227.4:4185 , piece_id=JOLBEJZMIC5UTTOGNS654EZVIYANB5UE6NLGJKOVP2JGJ3QYWCFQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.785974919 (499h3m29.657057964s) [noise=true , address=145.79.227.45:4230 , piece_id=27H4IQBNLB4DTQGH3MDE3P6WBBH7GKO33I3ZDWY25UGFDFPDNBXQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.791776337 (499h3m29.662859382s) [noise=true , address=145.79.227.46:4603 , piece_id=5XTSPOFXJ5WG77MVXSQVY53GRJ3M3CCD3GW3PT22SOX6QCJFDNHQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:30.037534750 (499h3m29.908617785s) [noise=true , address=45.91.116.7:4010 , piece_id=S6LZG5MC6A6NURSD5N2OF2X74DMCG5GN4CKB2ZPSVBSQGF6MUG4A, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:30.061182980 (499h3m29.932266015s) [noise=true , address=45.91.116.6:4975 , piece_id=ZXWQMTVELBYUB2FAYFDI2OEPHACFEZK5T3DXNZXFFH4RX7CL5EEQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:30.061271860 (499h3m29.932354895s) [seg_pos={0 0}, upload=438015]: Segment reached optimal threshold of 13 pieces. Dec 26 10:32:30.061580838 (499h3m29.932663883s) [noise=true , address=45.91.116.11:4120 , piece_id=3LTTNZIS3YPJ6OBKE3Q76ETFDVI2SUPESHBCLFVUH2CGUTLEZYCQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 10:32:30.061620123 (499h3m29.932703188s) [noise=true , address=45.91.116.11:4120 , piece_id=3LTTNZIS3YPJ6OBKE3Q76ETFDVI2SUPESHBCLFVUH2CGUTLEZYCQ, seg_pos={0 0}, upload=438015]: Long tail cancellation due to optimal threshold Dec 26 10:32:30.061682210 (499h3m29.932765256s) [noise=true , address=77.90.31.15:4700 , piece_id=O72QP657I6TF6A7U6OQLFA7YON4C2FSSJZP4NQYMHD464QNXQUMQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: read tcp 188.166.225.170:41482->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:32:30.061705164 (499h3m29.9327882s) [noise=true , address=77.90.31.15:4700 , piece_id=O72QP657I6TF6A7U6OQLFA7YON4C2FSSJZP4NQYMHD464QNXQUMQ, seg_pos={0 0}, upload=438015]: Long tail cancellation due to optimal threshold Dec 26 10:32:30.061763296 (499h3m29.932846331s) [noise=true , address=77.90.31.17:4770 , piece_id=NY52IIK2M7HQSRCYFLJROHKUZ5U5FPCPXGQGURWPPEJZASEPJYUQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:45072->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:45072->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45072->77.90.31.17:4770: use of closed network connection Dec 26 10:32:30.061770028 (499h3m29.932853063s) [noise=true , address=77.90.31.17:4770 , piece_id=NY52IIK2M7HQSRCYFLJROHKUZ5U5FPCPXGQGURWPPEJZASEPJYUQ, seg_pos={0 0}, upload=438015]: Long tail cancellation due to optimal threshold Dec 26 10:32:30.061785908 (499h3m29.932868953s) [seg_pos={0 0}, upload=438015]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:32:30.061895307 (499h3m29.932978352s) [upload=438015]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:32:30.061899966 (499h3m29.932983001s) [upload=438015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:32:30.345947669 (499h3m30.217030714s) [upload=438015]: Done uploading segments... Dec 26 10:32:30.345971524 (499h3m30.217054559s) [upload=438015]: Upload finished. err: <nil> Duration: 5.673444493s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.........X..................X.......XXX......X..........] duration: 44.595954066s trace: https://jaeger.dev.storj.io/trace/7e25338f73046b2a FAIL (2025-12-26 10:34:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033130872 (498h55m22.904213918s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033471341 (498h55m22.904554386s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.033892292 (498h55m22.904975328s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034387946 (498h55m22.905470991s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.034807164 (498h55m22.905890209s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035259836 (498h55m22.906342881s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.035727828 (498h55m22.906810863s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036033871 (498h55m22.907116906s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.036579810 (498h55m22.907662855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.037767882 (498h55m22.908850927s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.038064076 (498h55m22.909147122s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.039224646 (498h55m22.910307691s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040040249 (498h55m22.911123294s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.040366420 (498h55m22.911449455s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:23.041569130 (498h55m22.912652175s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Uploading piece... Dec 26 10:24:24.513239450 (498h55m24.384322485s) [upload=437904]: Next returned nil segment Dec 26 10:24:24.513254640 (498h55m24.384337685s) [upload=437904]: Waiting for error group managing segments... Dec 26 10:24:24.926029601 (498h55m24.797112636s) [noise=true , address=145.79.227.20:4459 , piece_id=KQDMSF5FEUN77FECCJDDH7F2YGG7ADWAIYHBPOVR7WI3DYJIVXAA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.027458408 (498h55m24.898541443s) [noise=true , address=64.49.13.170:4718 , piece_id=DDNMMWFJNVPS4UXAVLPYAIPBP4VELK5IC3LWHLBGIMM6YADENRPA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.094888666 (498h55m24.965971701s) [noise=true , address=77.90.31.13:4630 , piece_id=75BM5VIJUIF3NZFGJAXDTBEEC4BZFHFZH7QJKUER7IYRZTAQBDEQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.102988388 (498h55m24.974071433s) [noise=true , address=77.90.31.12:4624 , piece_id=5KXSW5FWSPBGA6T3I4DUSJL35PC27PSCWGWLXLMAP6SKQEE664IQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.109420056 (498h55m24.980503091s) [noise=true , address=77.90.31.15:4732 , piece_id=QDQXC2LEVGS7K5OHDX7Q3ZETJJ37XFXIS6BXXNFCUTKFEMDLJ23A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.122147140 (498h55m24.993230186s) [noise=true , address=45.91.116.7:4028 , piece_id=2KE7AFSSKMWNMPBXSXJ24ZWJOUTGBXFT5F7RE7DV3YD6QEUM4LHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.187517306 (498h55m25.058600351s) [noise=true , address=77.90.31.21:4892 , piece_id=KK4X7FVFFKJ6RJ5ZTHJCE4QRUTDGADZTW3OIME6FV7EOFFJA6J7Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.195537778 (498h55m25.066620823s) [noise=true , address=77.90.31.14:4671 , piece_id=ATYEZPFKZUNWSDSLRBDNJHH7GZHPTJSMLNSCSASDL6SFZVTE7UVQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.255475810 (498h55m25.126558855s) [noise=true , address=45.91.116.12:4242 , piece_id=55HC5BK66MIDGS3QWD2IHCEV3HSHN4FBABAFQEQQEVHATWV2L2PQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.546862739 (498h55m25.417945784s) [noise=true , address=77.90.31.18:4819 , piece_id=2QS3NB65Y3GQTNUCPVDRPVEUEEVE2O22O7BS75C2JZG54NBGTGSQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.792060605 (498h55m25.66314365s) [noise=true , address=145.79.227.40:4064 , piece_id=I7EFXYHOIVMB7NNEKNVYK4CLZ5PY7AQS6L7KLEZHMVV5ULN4U3DQ, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.809945911 (498h55m25.681028946s) [noise=true , address=145.79.227.44:4221 , piece_id=IZLW4HA64OMYW23ZBPBZU45GR4QCEREPUL76QL6RWWUA3LKKFN2A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.860843739 (498h55m25.731926774s) [noise=true , address=45.91.116.5:4302 , piece_id=JX6MFTVIEBLNYDR7VWKRPMYMSLU2TLOWXWSFAL54NUVG33SA3J4A, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861204966 (498h55m25.732288001s) [noise=true , address=145.79.227.30:4779 , piece_id=NYJP27UE6SNKDDPEI7XYGJJZG5TWXG7QII4JVWGOW5HSHT52ORTA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: <nil> Dec 26 10:24:25.861232289 (498h55m25.732315324s) [seg_pos={0 0}, upload=437904]: Segment reached optimal threshold of 14 pieces. Dec 26 10:24:25.861471414 (498h55m25.732554449s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861479399 (498h55m25.732562434s) [noise=true , address=77.90.31.19:4873 , piece_id=GGONAB7UC2XTOXDDRZO3X2S5YHSLB2Y4YHHUWVIBJJ6PFUHVKOHA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861580682 (498h55m25.732663717s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39532->64.49.13.169:4705: use of closed network connection Dec 26 10:24:25.861589008 (498h55m25.732672043s) [noise=true , address=64.49.13.169:4705 , piece_id=IZ5D63LQCR7O6X5FZGF426POFKVT3RZF7FKIU5I4PHYTUZPVQ24Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861840307 (498h55m25.732923342s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861847050 (498h55m25.732930085s) [noise=true , address=77.90.31.11:4574 , piece_id=3G7CKDS3UWNRRMU2IVFXIXCWOQY2HGCBTJ7MYWED2T2GVTWANF5Q, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861929887 (498h55m25.733012932s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: expected piece hash; context canceled; EOF Dec 26 10:24:25.861939285 (498h55m25.73302232s) [noise=true , address=45.91.116.9:4288 , piece_id=32HXGA3BHPKVENKCWFOB7UT5BATRTORYCKD6BI7A6AZRWDCSPADA, seg_pos={0 0}, upload=437904]: Long tail cancellation due to optimal threshold Dec 26 10:24:25.861959854 (498h55m25.733042889s) [seg_pos={0 0}, upload=437904]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 10:24:25.862033124 (498h55m25.733116159s) [upload=437904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:24:25.862037923 (498h55m25.733120958s) [upload=437904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:24:26.151364428 (498h55m26.022447463s) [upload=437904]: Done uploading segments... Dec 26 10:24:26.151393914 (498h55m26.022476949s) [upload=437904]: Upload finished. err: <nil> Duration: 44.595982522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X..] duration: 5.919180305s trace: https://jaeger.dev.storj.io/trace/6043e12ffff903ef FAIL (2025-12-26 10:34:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =438015]: Starting upload Dec 26 10:32:27.503304645 (499h3m27.37438768s) [upload=438015]: Done starting upload Dec 26 10:32:27.503555723 (499h3m27.374638768s) [upload=438015]: Uploading segments... Dec 26 10:32:27.503599195 (499h3m27.37468223s) [upload=438015]: Got next segment. Inline: false Dec 26 10:32:27.503644833 (499h3m27.374727868s) [upload=438015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:32:27.503648870 (499h3m27.374731905s) [upload=438015]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:32:27.745476991 (499h3m27.616560036s) [seg_pos={0 0}, upload=438015]: Begin upload segment... Dec 26 10:32:27.745536694 (499h3m27.616619739s) [seg_pos={0 0}, upload=438015]: Done begin upload segment. Dec 26 10:32:27.745564968 (499h3m27.616648003s) [noise=true , address=145.79.227.39:4014 , piece_id=XOZYZ6IW7VTHODKCE47BBH6TRCJBNZAIOQECIA7BYN6OF7MXKAOA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.745572803 (499h3m27.616655838s) [noise=true , address=145.79.227.15:4876 , piece_id=MC5RYVNACVGCQLJ3D3HK3QGNRZ6WMOYWEP2ELPJIAM2UNBZTZYXA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.746267105 (499h3m27.61735015s) [noise=true , address=45.91.116.6:4975 , piece_id=ZXWQMTVELBYUB2FAYFDI2OEPHACFEZK5T3DXNZXFFH4RX7CL5EEQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.746839796 (499h3m27.617922841s) [noise=true , address=145.79.227.38:4977 , piece_id=MNIKKIRFDNGHMK4YANINDCZSBRWOYSMBBD6375JGPYY3NBYSF3VA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.747186877 (499h3m27.618269912s) [noise=true , address=145.79.227.52:4446 , piece_id=QM5DJJEGJ4U44YTHMS4DUPJUOYX6TO7H24P2LE3EMNVEMEYLRM4A, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.747573212 (499h3m27.618656257s) [noise=true , address=145.79.227.6:4258 , piece_id=DN6GTGMTXK3I2XBC7AW425E4RCFXMJ37HYIFCNW7LLPIVRYJCKVQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.747937996 (499h3m27.619021031s) [noise=true , address=45.91.116.7:4010 , piece_id=S6LZG5MC6A6NURSD5N2OF2X74DMCG5GN4CKB2ZPSVBSQGF6MUG4A, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.748273225 (499h3m27.61935627s) [noise=true , address=145.79.227.13:4802 , piece_id=GKZKQCAD7PXCTOADWCX4ADN5JFFYQPGODHWN3WQGFC7HZKHUXL5A, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.749530719 (499h3m27.620613765s) [noise=true , address=145.79.227.29:4942 , piece_id=DPXUJQJ3BJZPYR3GXLT6GH3DUMUKOZQIBTUQHTKANNVKTG6RLUPQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.749561648 (499h3m27.620644694s) [noise=true , address=145.79.227.46:4603 , piece_id=5XTSPOFXJ5WG77MVXSQVY53GRJ3M3CCD3GW3PT22SOX6QCJFDNHQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.751321760 (499h3m27.622404806s) [noise=true , address=145.79.227.45:4230 , piece_id=27H4IQBNLB4DTQGH3MDE3P6WBBH7GKO33I3ZDWY25UGFDFPDNBXQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.753391351 (499h3m27.624474386s) [noise=true , address=77.90.31.17:4770 , piece_id=NY52IIK2M7HQSRCYFLJROHKUZ5U5FPCPXGQGURWPPEJZASEPJYUQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.755285377 (499h3m27.626368422s) [noise=true , address=45.91.116.11:4120 , piece_id=3LTTNZIS3YPJ6OBKE3Q76ETFDVI2SUPESHBCLFVUH2CGUTLEZYCQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.757601458 (499h3m27.628684503s) [noise=true , address=77.90.31.15:4700 , piece_id=O72QP657I6TF6A7U6OQLFA7YON4C2FSSJZP4NQYMHD464QNXQUMQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.759623167 (499h3m27.630706202s) [noise=true , address=145.79.227.4:4185 , piece_id=JOLBEJZMIC5UTTOGNS654EZVIYANB5UE6NLGJKOVP2JGJ3QYWCFQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.760142336 (499h3m27.631225371s) [noise=true , address=145.79.227.48:4301 , piece_id=G46SWJPXKAMMN6JZTBYK27TUBY7W4LUOXASJK36LQS6MAXF6E5XA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:29.467859286 (499h3m29.338942341s) [upload=438015]: Next returned nil segment Dec 26 10:32:29.467876819 (499h3m29.338959864s) [upload=438015]: Waiting for error group managing segments... Dec 26 10:32:29.756063633 (499h3m29.627146668s) [noise=true , address=145.79.227.38:4977 , piece_id=MNIKKIRFDNGHMK4YANINDCZSBRWOYSMBBD6375JGPYY3NBYSF3VA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.756671280 (499h3m29.627754325s) [noise=true , address=145.79.227.15:4876 , piece_id=MC5RYVNACVGCQLJ3D3HK3QGNRZ6WMOYWEP2ELPJIAM2UNBZTZYXA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.757161754 (499h3m29.628244799s) [noise=true , address=145.79.227.29:4942 , piece_id=DPXUJQJ3BJZPYR3GXLT6GH3DUMUKOZQIBTUQHTKANNVKTG6RLUPQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.757778659 (499h3m29.628861704s) [noise=true , address=145.79.227.13:4802 , piece_id=GKZKQCAD7PXCTOADWCX4ADN5JFFYQPGODHWN3WQGFC7HZKHUXL5A, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.757996383 (499h3m29.629079428s) [noise=true , address=145.79.227.52:4446 , piece_id=QM5DJJEGJ4U44YTHMS4DUPJUOYX6TO7H24P2LE3EMNVEMEYLRM4A, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.763864007 (499h3m29.634947052s) [noise=true , address=145.79.227.6:4258 , piece_id=DN6GTGMTXK3I2XBC7AW425E4RCFXMJ37HYIFCNW7LLPIVRYJCKVQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.764489287 (499h3m29.635572332s) [noise=true , address=145.79.227.48:4301 , piece_id=G46SWJPXKAMMN6JZTBYK27TUBY7W4LUOXASJK36LQS6MAXF6E5XA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.764947540 (499h3m29.636030585s) [noise=true , address=145.79.227.39:4014 , piece_id=XOZYZ6IW7VTHODKCE47BBH6TRCJBNZAIOQECIA7BYN6OF7MXKAOA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.780700537 (499h3m29.651783582s) [noise=true , address=145.79.227.4:4185 , piece_id=JOLBEJZMIC5UTTOGNS654EZVIYANB5UE6NLGJKOVP2JGJ3QYWCFQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.785974919 (499h3m29.657057964s) [noise=true , address=145.79.227.45:4230 , piece_id=27H4IQBNLB4DTQGH3MDE3P6WBBH7GKO33I3ZDWY25UGFDFPDNBXQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.791776337 (499h3m29.662859382s) [noise=true , address=145.79.227.46:4603 , piece_id=5XTSPOFXJ5WG77MVXSQVY53GRJ3M3CCD3GW3PT22SOX6QCJFDNHQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:30.037534750 (499h3m29.908617785s) [noise=true , address=45.91.116.7:4010 , piece_id=S6LZG5MC6A6NURSD5N2OF2X74DMCG5GN4CKB2ZPSVBSQGF6MUG4A, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:30.061182980 (499h3m29.932266015s) [noise=true , address=45.91.116.6:4975 , piece_id=ZXWQMTVELBYUB2FAYFDI2OEPHACFEZK5T3DXNZXFFH4RX7CL5EEQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:30.061271860 (499h3m29.932354895s) [seg_pos={0 0}, upload=438015]: Segment reached optimal threshold of 13 pieces. Dec 26 10:32:30.061580838 (499h3m29.932663883s) [noise=true , address=45.91.116.11:4120 , piece_id=3LTTNZIS3YPJ6OBKE3Q76ETFDVI2SUPESHBCLFVUH2CGUTLEZYCQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 10:32:30.061620123 (499h3m29.932703188s) [noise=true , address=45.91.116.11:4120 , piece_id=3LTTNZIS3YPJ6OBKE3Q76ETFDVI2SUPESHBCLFVUH2CGUTLEZYCQ, seg_pos={0 0}, upload=438015]: Long tail cancellation due to optimal threshold Dec 26 10:32:30.061682210 (499h3m29.932765256s) [noise=true , address=77.90.31.15:4700 , piece_id=O72QP657I6TF6A7U6OQLFA7YON4C2FSSJZP4NQYMHD464QNXQUMQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: read tcp 188.166.225.170:41482->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:32:30.061705164 (499h3m29.9327882s) [noise=true , address=77.90.31.15:4700 , piece_id=O72QP657I6TF6A7U6OQLFA7YON4C2FSSJZP4NQYMHD464QNXQUMQ, seg_pos={0 0}, upload=438015]: Long tail cancellation due to optimal threshold Dec 26 10:32:30.061763296 (499h3m29.932846331s) [noise=true , address=77.90.31.17:4770 , piece_id=NY52IIK2M7HQSRCYFLJROHKUZ5U5FPCPXGQGURWPPEJZASEPJYUQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:45072->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:45072->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45072->77.90.31.17:4770: use of closed network connection Dec 26 10:32:30.061770028 (499h3m29.932853063s) [noise=true , address=77.90.31.17:4770 , piece_id=NY52IIK2M7HQSRCYFLJROHKUZ5U5FPCPXGQGURWPPEJZASEPJYUQ, seg_pos={0 0}, upload=438015]: Long tail cancellation due to optimal threshold Dec 26 10:32:30.061785908 (499h3m29.932868953s) [seg_pos={0 0}, upload=438015]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:32:30.061895307 (499h3m29.932978352s) [upload=438015]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:32:30.061899966 (499h3m29.932983001s) [upload=438015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:32:30.345947669 (499h3m30.217030714s) [upload=438015]: Done uploading segments... Dec 26 10:32:30.345971524 (499h3m30.217054559s) [upload=438015]: Upload finished. err: <nil> Duration: 5.919213287s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.XXX.X..X.XXXXX.X.X..XX...X.X.XX....XX.......X.X.....X..X.] duration: 29.624995168s trace: https://jaeger.dev.storj.io/trace/6043e12ffff903ef FAIL (2025-12-26 10:33:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =438015]: Starting upload Dec 26 10:32:27.503304645 (499h3m27.37438768s) [upload=438015]: Done starting upload Dec 26 10:32:27.503555723 (499h3m27.374638768s) [upload=438015]: Uploading segments... Dec 26 10:32:27.503599195 (499h3m27.37468223s) [upload=438015]: Got next segment. Inline: false Dec 26 10:32:27.503644833 (499h3m27.374727868s) [upload=438015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 10:32:27.503648870 (499h3m27.374731905s) [upload=438015]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 10:32:27.745476991 (499h3m27.616560036s) [seg_pos={0 0}, upload=438015]: Begin upload segment... Dec 26 10:32:27.745536694 (499h3m27.616619739s) [seg_pos={0 0}, upload=438015]: Done begin upload segment. Dec 26 10:32:27.745564968 (499h3m27.616648003s) [noise=true , address=145.79.227.39:4014 , piece_id=XOZYZ6IW7VTHODKCE47BBH6TRCJBNZAIOQECIA7BYN6OF7MXKAOA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.745572803 (499h3m27.616655838s) [noise=true , address=145.79.227.15:4876 , piece_id=MC5RYVNACVGCQLJ3D3HK3QGNRZ6WMOYWEP2ELPJIAM2UNBZTZYXA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.746267105 (499h3m27.61735015s) [noise=true , address=45.91.116.6:4975 , piece_id=ZXWQMTVELBYUB2FAYFDI2OEPHACFEZK5T3DXNZXFFH4RX7CL5EEQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.746839796 (499h3m27.617922841s) [noise=true , address=145.79.227.38:4977 , piece_id=MNIKKIRFDNGHMK4YANINDCZSBRWOYSMBBD6375JGPYY3NBYSF3VA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.747186877 (499h3m27.618269912s) [noise=true , address=145.79.227.52:4446 , piece_id=QM5DJJEGJ4U44YTHMS4DUPJUOYX6TO7H24P2LE3EMNVEMEYLRM4A, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.747573212 (499h3m27.618656257s) [noise=true , address=145.79.227.6:4258 , piece_id=DN6GTGMTXK3I2XBC7AW425E4RCFXMJ37HYIFCNW7LLPIVRYJCKVQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.747937996 (499h3m27.619021031s) [noise=true , address=45.91.116.7:4010 , piece_id=S6LZG5MC6A6NURSD5N2OF2X74DMCG5GN4CKB2ZPSVBSQGF6MUG4A, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.748273225 (499h3m27.61935627s) [noise=true , address=145.79.227.13:4802 , piece_id=GKZKQCAD7PXCTOADWCX4ADN5JFFYQPGODHWN3WQGFC7HZKHUXL5A, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.749530719 (499h3m27.620613765s) [noise=true , address=145.79.227.29:4942 , piece_id=DPXUJQJ3BJZPYR3GXLT6GH3DUMUKOZQIBTUQHTKANNVKTG6RLUPQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.749561648 (499h3m27.620644694s) [noise=true , address=145.79.227.46:4603 , piece_id=5XTSPOFXJ5WG77MVXSQVY53GRJ3M3CCD3GW3PT22SOX6QCJFDNHQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.751321760 (499h3m27.622404806s) [noise=true , address=145.79.227.45:4230 , piece_id=27H4IQBNLB4DTQGH3MDE3P6WBBH7GKO33I3ZDWY25UGFDFPDNBXQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.753391351 (499h3m27.624474386s) [noise=true , address=77.90.31.17:4770 , piece_id=NY52IIK2M7HQSRCYFLJROHKUZ5U5FPCPXGQGURWPPEJZASEPJYUQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.755285377 (499h3m27.626368422s) [noise=true , address=45.91.116.11:4120 , piece_id=3LTTNZIS3YPJ6OBKE3Q76ETFDVI2SUPESHBCLFVUH2CGUTLEZYCQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.757601458 (499h3m27.628684503s) [noise=true , address=77.90.31.15:4700 , piece_id=O72QP657I6TF6A7U6OQLFA7YON4C2FSSJZP4NQYMHD464QNXQUMQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.759623167 (499h3m27.630706202s) [noise=true , address=145.79.227.4:4185 , piece_id=JOLBEJZMIC5UTTOGNS654EZVIYANB5UE6NLGJKOVP2JGJ3QYWCFQ, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:27.760142336 (499h3m27.631225371s) [noise=true , address=145.79.227.48:4301 , piece_id=G46SWJPXKAMMN6JZTBYK27TUBY7W4LUOXASJK36LQS6MAXF6E5XA, seg_pos={0 0}, upload=438015]: Uploading piece... Dec 26 10:32:29.467859286 (499h3m29.338942341s) [upload=438015]: Next returned nil segment Dec 26 10:32:29.467876819 (499h3m29.338959864s) [upload=438015]: Waiting for error group managing segments... Dec 26 10:32:29.756063633 (499h3m29.627146668s) [noise=true , address=145.79.227.38:4977 , piece_id=MNIKKIRFDNGHMK4YANINDCZSBRWOYSMBBD6375JGPYY3NBYSF3VA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.756671280 (499h3m29.627754325s) [noise=true , address=145.79.227.15:4876 , piece_id=MC5RYVNACVGCQLJ3D3HK3QGNRZ6WMOYWEP2ELPJIAM2UNBZTZYXA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.757161754 (499h3m29.628244799s) [noise=true , address=145.79.227.29:4942 , piece_id=DPXUJQJ3BJZPYR3GXLT6GH3DUMUKOZQIBTUQHTKANNVKTG6RLUPQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.757778659 (499h3m29.628861704s) [noise=true , address=145.79.227.13:4802 , piece_id=GKZKQCAD7PXCTOADWCX4ADN5JFFYQPGODHWN3WQGFC7HZKHUXL5A, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.757996383 (499h3m29.629079428s) [noise=true , address=145.79.227.52:4446 , piece_id=QM5DJJEGJ4U44YTHMS4DUPJUOYX6TO7H24P2LE3EMNVEMEYLRM4A, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.763864007 (499h3m29.634947052s) [noise=true , address=145.79.227.6:4258 , piece_id=DN6GTGMTXK3I2XBC7AW425E4RCFXMJ37HYIFCNW7LLPIVRYJCKVQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.764489287 (499h3m29.635572332s) [noise=true , address=145.79.227.48:4301 , piece_id=G46SWJPXKAMMN6JZTBYK27TUBY7W4LUOXASJK36LQS6MAXF6E5XA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.764947540 (499h3m29.636030585s) [noise=true , address=145.79.227.39:4014 , piece_id=XOZYZ6IW7VTHODKCE47BBH6TRCJBNZAIOQECIA7BYN6OF7MXKAOA, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.780700537 (499h3m29.651783582s) [noise=true , address=145.79.227.4:4185 , piece_id=JOLBEJZMIC5UTTOGNS654EZVIYANB5UE6NLGJKOVP2JGJ3QYWCFQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.785974919 (499h3m29.657057964s) [noise=true , address=145.79.227.45:4230 , piece_id=27H4IQBNLB4DTQGH3MDE3P6WBBH7GKO33I3ZDWY25UGFDFPDNBXQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:29.791776337 (499h3m29.662859382s) [noise=true , address=145.79.227.46:4603 , piece_id=5XTSPOFXJ5WG77MVXSQVY53GRJ3M3CCD3GW3PT22SOX6QCJFDNHQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:30.037534750 (499h3m29.908617785s) [noise=true , address=45.91.116.7:4010 , piece_id=S6LZG5MC6A6NURSD5N2OF2X74DMCG5GN4CKB2ZPSVBSQGF6MUG4A, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:30.061182980 (499h3m29.932266015s) [noise=true , address=45.91.116.6:4975 , piece_id=ZXWQMTVELBYUB2FAYFDI2OEPHACFEZK5T3DXNZXFFH4RX7CL5EEQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: <nil> Dec 26 10:32:30.061271860 (499h3m29.932354895s) [seg_pos={0 0}, upload=438015]: Segment reached optimal threshold of 13 pieces. Dec 26 10:32:30.061580838 (499h3m29.932663883s) [noise=true , address=45.91.116.11:4120 , piece_id=3LTTNZIS3YPJ6OBKE3Q76ETFDVI2SUPESHBCLFVUH2CGUTLEZYCQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 10:32:30.061620123 (499h3m29.932703188s) [noise=true , address=45.91.116.11:4120 , piece_id=3LTTNZIS3YPJ6OBKE3Q76ETFDVI2SUPESHBCLFVUH2CGUTLEZYCQ, seg_pos={0 0}, upload=438015]: Long tail cancellation due to optimal threshold Dec 26 10:32:30.061682210 (499h3m29.932765256s) [noise=true , address=77.90.31.15:4700 , piece_id=O72QP657I6TF6A7U6OQLFA7YON4C2FSSJZP4NQYMHD464QNXQUMQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: read tcp 188.166.225.170:41482->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 10:32:30.061705164 (499h3m29.9327882s) [noise=true , address=77.90.31.15:4700 , piece_id=O72QP657I6TF6A7U6OQLFA7YON4C2FSSJZP4NQYMHD464QNXQUMQ, seg_pos={0 0}, upload=438015]: Long tail cancellation due to optimal threshold Dec 26 10:32:30.061763296 (499h3m29.932846331s) [noise=true , address=77.90.31.17:4770 , piece_id=NY52IIK2M7HQSRCYFLJROHKUZ5U5FPCPXGQGURWPPEJZASEPJYUQ, seg_pos={0 0}, upload=438015]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:45072->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:45072->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45072->77.90.31.17:4770: use of closed network connection Dec 26 10:32:30.061770028 (499h3m29.932853063s) [noise=true , address=77.90.31.17:4770 , piece_id=NY52IIK2M7HQSRCYFLJROHKUZ5U5FPCPXGQGURWPPEJZASEPJYUQ, seg_pos={0 0}, upload=438015]: Long tail cancellation due to optimal threshold Dec 26 10:32:30.061785908 (499h3m29.932868953s) [seg_pos={0 0}, upload=438015]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 10:32:30.061895307 (499h3m29.932978352s) [upload=438015]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 10:32:30.061899966 (499h3m29.932983001s) [upload=438015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 10:32:30.345947669 (499h3m30.217030714s) [upload=438015]: Done uploading segments... Dec 26 10:32:30.345971524 (499h3m30.217054559s) [upload=438015]: Upload finished. err: <nil> Duration: 29.62502206s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)