Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X] duration: 30.003871602s trace: https://jaeger.dev.storj.io/trace/6390b17b5edf4ccc FAIL (2025-12-27 04:57:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 04:57:28.726777563 (517h28m28.597860598s) [upload=454018]: Starting upload Dec 27 04:57:28.726874217 (517h28m28.597957262s) [upload=454018]: Done starting upload Dec 27 04:57:28.727000007 (517h28m28.598083052s) [upload=454018]: Uploading segments... Dec 27 04:57:28.727237108 (517h28m28.598320153s) [upload=454018]: Got next segment. Inline: false Dec 27 04:57:28.727279439 (517h28m28.598362474s) [upload=454018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:57:28.727285130 (517h28m28.598368165s) [upload=454018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:57:28.983209413 (517h28m28.854292468s) [seg_pos={0 0}, upload=454018]: Begin upload segment... Dec 27 04:57:28.983262725 (517h28m28.85434577s) [seg_pos={0 0}, upload=454018]: Done begin upload segment. Dec 27 04:57:28.983296289 (517h28m28.854379334s) [noise=true , address=45.91.116.14:4192 , piece_id=25VV4LWEAHW6IV7DZYKMD7QSIJHCNTT6NAHC64JG3FS64LK754PA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.983334321 (517h28m28.854417366s) [noise=true , address=145.79.227.42:4121 , piece_id=NITUZXHONLJZV7ABSUXOBOFJP4FWIZBPIOTMXKR7XAOY7CJYUC6Q, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.984019315 (517h28m28.85510235s) [noise=true , address=145.79.227.15:4874 , piece_id=XUTKF5G5DMNIRKEVQODAVRWLBW26NRVE5ZSVBFP6RZDI63POZC2A, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.984079700 (517h28m28.855162745s) [noise=true , address=145.79.227.11:4729 , piece_id=OUJQBQVK62YAW4Y57B3VUVLY3WX6HO4PAGV56SMZRNTCXYFEX37Q, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.984630639 (517h28m28.855713674s) [noise=true , address=145.79.227.36:4519 , piece_id=SYCUWEDK3NWWBY6AWJCRUUF3W6BT7YUTFRJK52PUTLABNPE22LGQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.985234529 (517h28m28.856317574s) [noise=true , address=45.91.116.7:4009 , piece_id=AVHNEBFAHCDL5G5KWOS7KAPIYXAMHR2CMOKIGOOZAGBZQS2AFTSA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.985608100 (517h28m28.856691136s) [noise=true , address=145.79.227.10:4403 , piece_id=TKP2W5Q35SVHFSMFNL6ZLF2SLWIOUD6G2C6C4IYOT6CJZHMIMKZQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.985825514 (517h28m28.856908559s) [noise=true , address=145.79.227.38:4979 , piece_id=FIVEH7ZHEBUH22P2K4UJB7E4TNJVKDYDLPO2WKFXXF35W2QXR2HA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.986017941 (517h28m28.857100976s) [noise=true , address=77.90.31.18:4808 , piece_id=O5MCFVIUSYB725F4QH6JARKMXOSQAXSMEXW36SJBGWFBLP4WX3IQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.986361275 (517h28m28.85744432s) [noise=true , address=145.79.227.45:4230 , piece_id=5FQGPG6WJTEXZ5TFWJSDEK7CGXH27YBXW44XRNJJWDOL3O72OJ4A, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.986373278 (517h28m28.857456313s) [noise=true , address=45.91.116.11:4120 , piece_id=CNCVXNGV32IN3C5FLBNNQSDYWHNIJQBBYJU4PMXPLXSNZ76GG7PQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.986824908 (517h28m28.857907943s) [noise=true , address=45.91.116.5:4298 , piece_id=FNTZRWTBJNRNRXV3TWBHKC3FHDUYVALVIUOUBBY3WEGBNSLZDK3A, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.987025450 (517h28m28.858108495s) [noise=true , address=77.90.31.14:4664 , piece_id=HMZNH5YGI77YMITYH7PI5NRWXBK5MYKF4IPN7FWKXA2XJYG647KQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.987163753 (517h28m28.858246798s) [noise=true , address=145.79.227.13:4802 , piece_id=SBGS2QNO54CJSIZKT6RRVKXYOL32J2CPIATHKABOJST5YCOCEQRA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.987737024 (517h28m28.858820059s) [noise=true , address=145.79.227.46:4601 , piece_id=67MAU6L6IF2DULXQTSWXGCDI5SJWXM4A75CLNWRLOPWBI62R62IA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.987800986 (517h28m28.858884031s) [noise=true , address=77.90.31.17:4770 , piece_id=TSZKXNQZ2MXTKCPR6VLNMFVMRLETL2Y5EWZ57MCTK4RBQTZU4D7A, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:30.599796542 (517h28m30.470879577s) [upload=454018]: Next returned nil segment Dec 27 04:57:30.599812842 (517h28m30.470895877s) [upload=454018]: Waiting for error group managing segments... Dec 27 04:57:30.985521317 (517h28m30.856604353s) [noise=true , address=145.79.227.42:4121 , piece_id=NITUZXHONLJZV7ABSUXOBOFJP4FWIZBPIOTMXKR7XAOY7CJYUC6Q, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:30.997362936 (517h28m30.868445971s) [noise=true , address=145.79.227.36:4519 , piece_id=SYCUWEDK3NWWBY6AWJCRUUF3W6BT7YUTFRJK52PUTLABNPE22LGQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.002520557 (517h28m30.873603592s) [noise=true , address=145.79.227.38:4979 , piece_id=FIVEH7ZHEBUH22P2K4UJB7E4TNJVKDYDLPO2WKFXXF35W2QXR2HA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.005987178 (517h28m30.877070223s) [noise=true , address=145.79.227.46:4601 , piece_id=67MAU6L6IF2DULXQTSWXGCDI5SJWXM4A75CLNWRLOPWBI62R62IA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.009730596 (517h28m30.880813641s) [noise=true , address=145.79.227.11:4729 , piece_id=OUJQBQVK62YAW4Y57B3VUVLY3WX6HO4PAGV56SMZRNTCXYFEX37Q, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.009782414 (517h28m30.880865459s) [noise=true , address=145.79.227.15:4874 , piece_id=XUTKF5G5DMNIRKEVQODAVRWLBW26NRVE5ZSVBFP6RZDI63POZC2A, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.015590273 (517h28m30.886673308s) [noise=true , address=145.79.227.10:4403 , piece_id=TKP2W5Q35SVHFSMFNL6ZLF2SLWIOUD6G2C6C4IYOT6CJZHMIMKZQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.020506705 (517h28m30.89158974s) [noise=true , address=145.79.227.45:4230 , piece_id=5FQGPG6WJTEXZ5TFWJSDEK7CGXH27YBXW44XRNJJWDOL3O72OJ4A, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.020525381 (517h28m30.891608516s) [noise=true , address=145.79.227.13:4802 , piece_id=SBGS2QNO54CJSIZKT6RRVKXYOL32J2CPIATHKABOJST5YCOCEQRA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.239196067 (517h28m31.110279112s) [noise=true , address=77.90.31.18:4808 , piece_id=O5MCFVIUSYB725F4QH6JARKMXOSQAXSMEXW36SJBGWFBLP4WX3IQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.265170878 (517h28m31.136253923s) [noise=true , address=77.90.31.17:4770 , piece_id=TSZKXNQZ2MXTKCPR6VLNMFVMRLETL2Y5EWZ57MCTK4RBQTZU4D7A, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.269304979 (517h28m31.140388024s) [noise=true , address=77.90.31.14:4664 , piece_id=HMZNH5YGI77YMITYH7PI5NRWXBK5MYKF4IPN7FWKXA2XJYG647KQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.312355986 (517h28m31.183439031s) [noise=true , address=45.91.116.7:4009 , piece_id=AVHNEBFAHCDL5G5KWOS7KAPIYXAMHR2CMOKIGOOZAGBZQS2AFTSA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.312374862 (517h28m31.183457897s) [seg_pos={0 0}, upload=454018]: Segment reached optimal threshold of 13 pieces. Dec 27 04:57:31.312611743 (517h28m31.183694778s) [noise=true , address=45.91.116.11:4120 , piece_id=CNCVXNGV32IN3C5FLBNNQSDYWHNIJQBBYJU4PMXPLXSNZ76GG7PQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 04:57:31.312617914 (517h28m31.18370095s) [noise=true , address=45.91.116.11:4120 , piece_id=CNCVXNGV32IN3C5FLBNNQSDYWHNIJQBBYJU4PMXPLXSNZ76GG7PQ, seg_pos={0 0}, upload=454018]: Long tail cancellation due to optimal threshold Dec 27 04:57:31.312669383 (517h28m31.183752418s) [noise=true , address=45.91.116.5:4298 , piece_id=FNTZRWTBJNRNRXV3TWBHKC3FHDUYVALVIUOUBBY3WEGBNSLZDK3A, seg_pos={0 0}, upload=454018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:52858->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:57:31.312687627 (517h28m31.183770662s) [noise=true , address=45.91.116.5:4298 , piece_id=FNTZRWTBJNRNRXV3TWBHKC3FHDUYVALVIUOUBBY3WEGBNSLZDK3A, seg_pos={0 0}, upload=454018]: Long tail cancellation due to optimal threshold Dec 27 04:57:31.312744526 (517h28m31.183827571s) [noise=true , address=45.91.116.14:4192 , piece_id=25VV4LWEAHW6IV7DZYKMD7QSIJHCNTT6NAHC64JG3FS64LK754PA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 27 04:57:31.312750507 (517h28m31.183833542s) [noise=true , address=45.91.116.14:4192 , piece_id=25VV4LWEAHW6IV7DZYKMD7QSIJHCNTT6NAHC64JG3FS64LK754PA, seg_pos={0 0}, upload=454018]: Long tail cancellation due to optimal threshold Dec 27 04:57:31.312766729 (517h28m31.183849764s) [seg_pos={0 0}, upload=454018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:57:31.312830609 (517h28m31.183913654s) [upload=454018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:57:31.312834989 (517h28m31.183918024s) [upload=454018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:57:31.612010370 (517h28m31.483093425s) [upload=454018]: Done uploading segments... Dec 27 04:57:31.612035938 (517h28m31.483118983s) [upload=454018]: Upload finished. err: <nil> Duration: 30.003893703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X......................X..........] duration: 37.788932991s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:57:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 37.788947779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.507762088s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:57:48 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.507784852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X......................X.........] duration: 42.028824034s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:57:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 42.028854191s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.499556463s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:56:45 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.499583856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..] duration: 6.282379694s trace: https://jaeger.dev.storj.io/trace/69f0122fd61b9ab6 FAIL (2025-12-27 04:56:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ) context deadline exceeded ======LOGS====== Dec 27 04:54:53.228016161 (517h25m53.099099206s) [upload=453981]: Starting upload Dec 27 04:54:53.228085944 (517h25m53.099168989s) [upload=453981]: Done starting upload Dec 27 04:54:53.228222013 (517h25m53.099305058s) [upload=453981]: Uploading segments... Dec 27 04:54:53.228328165 (517h25m53.09941121s) [upload=453981]: Got next segment. Inline: false Dec 27 04:54:53.228364304 (517h25m53.099447349s) [upload=453981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:54:53.228369984 (517h25m53.099453029s) [upload=453981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:54:53.471176480 (517h25m53.342259515s) [seg_pos={0 0}, upload=453981]: Begin upload segment... Dec 27 04:54:53.471236604 (517h25m53.342319639s) [seg_pos={0 0}, upload=453981]: Done begin upload segment. Dec 27 04:54:53.471278964 (517h25m53.342361999s) [noise=true , address=145.79.227.10:4403 , piece_id=QJT2FXUJCWMPC5BEEU6INHKZEGCR2QGXLDW5YO55BPIIOU7REU3Q, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.471300586 (517h25m53.342383621s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.471799074 (517h25m53.34288211s) [noise=true , address=77.90.31.12:4590 , piece_id=W2JYHHHVVGT2D27ZYZA3HHKLO7Z5ATUAKJGR5Q5MPHVAZHGMAWWA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.472138040 (517h25m53.343221085s) [noise=true , address=145.79.227.2:4115 , piece_id=VHT5DS4Z5NJSNTNVGEERXQTAUKAUIHI7TOWPQBMZCN3QVFQQXSFQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473034377 (517h25m53.344117412s) [noise=true , address=145.79.227.45:4229 , piece_id=ILHC3BM7TDMHUS4ACIDFSZHPIYW7XPXELKJWP5CGO5AEE2737DYQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473402668 (517h25m53.344485703s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473761913 (517h25m53.344844948s) [noise=true , address=145.79.227.15:4874 , piece_id=R2MHZBZ63W47CYGDE3D7LPXSYLK4LTFJINUI3PJCBX3HKRIABK2A, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.474149651 (517h25m53.345232696s) [noise=true , address=45.91.116.5:4297 , piece_id=YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.475456570 (517h25m53.346539605s) [noise=true , address=77.90.31.21:4880 , piece_id=FVVU2G2JKJ6DTYHT62NAU7SM7M5C2NDERINEDXBTQH2PZTZAVVUQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.475590674 (517h25m53.346673719s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.476650784 (517h25m53.347733819s) [noise=true , address=77.90.31.11:4554 , piece_id=7A24GXKVWBS6KMNAQY5LEFQPS34FJVMUTSBWOCTAN4M24HHB3BCA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.477108996 (517h25m53.348192041s) [noise=true , address=145.79.227.11:4730 , piece_id=GAPDONNT4DIA5YW73JWWJKKNXLDC5UVIYD3IPMMV5OI3GRO3PBCA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.478711297 (517h25m53.349794332s) [noise=true , address=45.91.116.13:4154 , piece_id=5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.480251329 (517h25m53.351334374s) [noise=true , address=145.79.227.23:4546 , piece_id=HWOYTQHNDDXXSET3DLLUKPB45NWJJWHRQ2RQQRY47E2HLEJRQBLA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.481777846 (517h25m53.352860881s) [noise=true , address=145.79.227.49:4337 , piece_id=H5D75YGDK6SBGCLYHPN2GTQXPFT46SLUPYNJB3AMCRDF2V5NC4KA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.483148365 (517h25m53.3542314s) [noise=true , address=145.79.227.35:4483 , piece_id=5GWKPQ2SPRO6JUS5AH5K3OJMLVK4U5UQMPDKXUN5NPYZXJT656EA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:55.192400484 (517h25m55.063483519s) [upload=453981]: Next returned nil segment Dec 27 04:54:55.192418167 (517h25m55.063501202s) [upload=453981]: Waiting for error group managing segments... Dec 27 04:54:55.477444407 (517h25m55.348527452s) [noise=true , address=145.79.227.15:4874 , piece_id=R2MHZBZ63W47CYGDE3D7LPXSYLK4LTFJINUI3PJCBX3HKRIABK2A, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.480670369 (517h25m55.351753414s) [noise=true , address=145.79.227.10:4403 , piece_id=QJT2FXUJCWMPC5BEEU6INHKZEGCR2QGXLDW5YO55BPIIOU7REU3Q, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.481618815 (517h25m55.35270187s) [noise=true , address=145.79.227.23:4546 , piece_id=HWOYTQHNDDXXSET3DLLUKPB45NWJJWHRQ2RQQRY47E2HLEJRQBLA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.494611344 (517h25m55.365694389s) [noise=true , address=145.79.227.2:4115 , piece_id=VHT5DS4Z5NJSNTNVGEERXQTAUKAUIHI7TOWPQBMZCN3QVFQQXSFQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.499504542 (517h25m55.370587577s) [noise=true , address=145.79.227.35:4483 , piece_id=5GWKPQ2SPRO6JUS5AH5K3OJMLVK4U5UQMPDKXUN5NPYZXJT656EA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.504255118 (517h25m55.375338153s) [noise=true , address=145.79.227.45:4229 , piece_id=ILHC3BM7TDMHUS4ACIDFSZHPIYW7XPXELKJWP5CGO5AEE2737DYQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.508654526 (517h25m55.379737561s) [noise=true , address=145.79.227.11:4730 , piece_id=GAPDONNT4DIA5YW73JWWJKKNXLDC5UVIYD3IPMMV5OI3GRO3PBCA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.509571982 (517h25m55.380655027s) [noise=true , address=145.79.227.49:4337 , piece_id=H5D75YGDK6SBGCLYHPN2GTQXPFT46SLUPYNJB3AMCRDF2V5NC4KA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.664817878 (517h25m55.535900913s) [noise=true , address=77.90.31.12:4590 , piece_id=W2JYHHHVVGT2D27ZYZA3HHKLO7Z5ATUAKJGR5Q5MPHVAZHGMAWWA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.674947937 (517h25m55.546030982s) [noise=true , address=77.90.31.11:4554 , piece_id=7A24GXKVWBS6KMNAQY5LEFQPS34FJVMUTSBWOCTAN4M24HHB3BCA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.774217743 (517h25m55.645300788s) [noise=true , address=77.90.31.21:4880 , piece_id=FVVU2G2JKJ6DTYHT62NAU7SM7M5C2NDERINEDXBTQH2PZTZAVVUQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.836321479 (517h25m55.707404524s) [noise=true , address=45.91.116.5:4297 , piece_id=YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:56.479032291 (517h25m56.350115336s) [noise=true , address=45.91.116.13:4154 , piece_id=5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:56.479057088 (517h25m56.350140123s) [seg_pos={0 0}, upload=453981]: Segment reached optimal threshold of 13 pieces. Dec 27 04:54:56.479269112 (517h25m56.350352157s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479278970 (517h25m56.350362005s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479391665 (517h25m56.3504747s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479407575 (517h25m56.35049061s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479491635 (517h25m56.35057467s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479501274 (517h25m56.350584319s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479548243 (517h25m56.350631278s) [seg_pos={0 0}, upload=453981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:54:56.479628085 (517h25m56.35071113s) [upload=453981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:54:56.479633856 (517h25m56.350716901s) [upload=453981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:54:56.771755175 (517h25m56.642838211s) [upload=453981]: Done uploading segments... Dec 27 04:54:56.771779040 (517h25m56.642862075s) [upload=453981]: Upload finished. err: <nil> Duration: 6.282410013s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.731228407s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:56:03 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.731257423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X......................X........] duration: 49.400434567s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:56:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 49.40045755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX.] duration: 22.109102369s trace: https://jaeger.dev.storj.io/trace/69f0122fd61b9ab6 FAIL (2025-12-27 04:55:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ) context deadline exceeded ======LOGS====== Dec 27 04:54:53.228016161 (517h25m53.099099206s) [upload=453981]: Starting upload Dec 27 04:54:53.228085944 (517h25m53.099168989s) [upload=453981]: Done starting upload Dec 27 04:54:53.228222013 (517h25m53.099305058s) [upload=453981]: Uploading segments... Dec 27 04:54:53.228328165 (517h25m53.09941121s) [upload=453981]: Got next segment. Inline: false Dec 27 04:54:53.228364304 (517h25m53.099447349s) [upload=453981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:54:53.228369984 (517h25m53.099453029s) [upload=453981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:54:53.471176480 (517h25m53.342259515s) [seg_pos={0 0}, upload=453981]: Begin upload segment... Dec 27 04:54:53.471236604 (517h25m53.342319639s) [seg_pos={0 0}, upload=453981]: Done begin upload segment. Dec 27 04:54:53.471278964 (517h25m53.342361999s) [noise=true , address=145.79.227.10:4403 , piece_id=QJT2FXUJCWMPC5BEEU6INHKZEGCR2QGXLDW5YO55BPIIOU7REU3Q, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.471300586 (517h25m53.342383621s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.471799074 (517h25m53.34288211s) [noise=true , address=77.90.31.12:4590 , piece_id=W2JYHHHVVGT2D27ZYZA3HHKLO7Z5ATUAKJGR5Q5MPHVAZHGMAWWA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.472138040 (517h25m53.343221085s) [noise=true , address=145.79.227.2:4115 , piece_id=VHT5DS4Z5NJSNTNVGEERXQTAUKAUIHI7TOWPQBMZCN3QVFQQXSFQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473034377 (517h25m53.344117412s) [noise=true , address=145.79.227.45:4229 , piece_id=ILHC3BM7TDMHUS4ACIDFSZHPIYW7XPXELKJWP5CGO5AEE2737DYQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473402668 (517h25m53.344485703s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473761913 (517h25m53.344844948s) [noise=true , address=145.79.227.15:4874 , piece_id=R2MHZBZ63W47CYGDE3D7LPXSYLK4LTFJINUI3PJCBX3HKRIABK2A, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.474149651 (517h25m53.345232696s) [noise=true , address=45.91.116.5:4297 , piece_id=YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.475456570 (517h25m53.346539605s) [noise=true , address=77.90.31.21:4880 , piece_id=FVVU2G2JKJ6DTYHT62NAU7SM7M5C2NDERINEDXBTQH2PZTZAVVUQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.475590674 (517h25m53.346673719s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.476650784 (517h25m53.347733819s) [noise=true , address=77.90.31.11:4554 , piece_id=7A24GXKVWBS6KMNAQY5LEFQPS34FJVMUTSBWOCTAN4M24HHB3BCA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.477108996 (517h25m53.348192041s) [noise=true , address=145.79.227.11:4730 , piece_id=GAPDONNT4DIA5YW73JWWJKKNXLDC5UVIYD3IPMMV5OI3GRO3PBCA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.478711297 (517h25m53.349794332s) [noise=true , address=45.91.116.13:4154 , piece_id=5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.480251329 (517h25m53.351334374s) [noise=true , address=145.79.227.23:4546 , piece_id=HWOYTQHNDDXXSET3DLLUKPB45NWJJWHRQ2RQQRY47E2HLEJRQBLA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.481777846 (517h25m53.352860881s) [noise=true , address=145.79.227.49:4337 , piece_id=H5D75YGDK6SBGCLYHPN2GTQXPFT46SLUPYNJB3AMCRDF2V5NC4KA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.483148365 (517h25m53.3542314s) [noise=true , address=145.79.227.35:4483 , piece_id=5GWKPQ2SPRO6JUS5AH5K3OJMLVK4U5UQMPDKXUN5NPYZXJT656EA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:55.192400484 (517h25m55.063483519s) [upload=453981]: Next returned nil segment Dec 27 04:54:55.192418167 (517h25m55.063501202s) [upload=453981]: Waiting for error group managing segments... Dec 27 04:54:55.477444407 (517h25m55.348527452s) [noise=true , address=145.79.227.15:4874 , piece_id=R2MHZBZ63W47CYGDE3D7LPXSYLK4LTFJINUI3PJCBX3HKRIABK2A, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.480670369 (517h25m55.351753414s) [noise=true , address=145.79.227.10:4403 , piece_id=QJT2FXUJCWMPC5BEEU6INHKZEGCR2QGXLDW5YO55BPIIOU7REU3Q, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.481618815 (517h25m55.35270187s) [noise=true , address=145.79.227.23:4546 , piece_id=HWOYTQHNDDXXSET3DLLUKPB45NWJJWHRQ2RQQRY47E2HLEJRQBLA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.494611344 (517h25m55.365694389s) [noise=true , address=145.79.227.2:4115 , piece_id=VHT5DS4Z5NJSNTNVGEERXQTAUKAUIHI7TOWPQBMZCN3QVFQQXSFQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.499504542 (517h25m55.370587577s) [noise=true , address=145.79.227.35:4483 , piece_id=5GWKPQ2SPRO6JUS5AH5K3OJMLVK4U5UQMPDKXUN5NPYZXJT656EA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.504255118 (517h25m55.375338153s) [noise=true , address=145.79.227.45:4229 , piece_id=ILHC3BM7TDMHUS4ACIDFSZHPIYW7XPXELKJWP5CGO5AEE2737DYQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.508654526 (517h25m55.379737561s) [noise=true , address=145.79.227.11:4730 , piece_id=GAPDONNT4DIA5YW73JWWJKKNXLDC5UVIYD3IPMMV5OI3GRO3PBCA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.509571982 (517h25m55.380655027s) [noise=true , address=145.79.227.49:4337 , piece_id=H5D75YGDK6SBGCLYHPN2GTQXPFT46SLUPYNJB3AMCRDF2V5NC4KA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.664817878 (517h25m55.535900913s) [noise=true , address=77.90.31.12:4590 , piece_id=W2JYHHHVVGT2D27ZYZA3HHKLO7Z5ATUAKJGR5Q5MPHVAZHGMAWWA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.674947937 (517h25m55.546030982s) [noise=true , address=77.90.31.11:4554 , piece_id=7A24GXKVWBS6KMNAQY5LEFQPS34FJVMUTSBWOCTAN4M24HHB3BCA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.774217743 (517h25m55.645300788s) [noise=true , address=77.90.31.21:4880 , piece_id=FVVU2G2JKJ6DTYHT62NAU7SM7M5C2NDERINEDXBTQH2PZTZAVVUQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.836321479 (517h25m55.707404524s) [noise=true , address=45.91.116.5:4297 , piece_id=YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:56.479032291 (517h25m56.350115336s) [noise=true , address=45.91.116.13:4154 , piece_id=5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:56.479057088 (517h25m56.350140123s) [seg_pos={0 0}, upload=453981]: Segment reached optimal threshold of 13 pieces. Dec 27 04:54:56.479269112 (517h25m56.350352157s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479278970 (517h25m56.350362005s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479391665 (517h25m56.3504747s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479407575 (517h25m56.35049061s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479491635 (517h25m56.35057467s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479501274 (517h25m56.350584319s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479548243 (517h25m56.350631278s) [seg_pos={0 0}, upload=453981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:54:56.479628085 (517h25m56.35071113s) [upload=453981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:54:56.479633856 (517h25m56.350716901s) [upload=453981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:54:56.771755175 (517h25m56.642838211s) [upload=453981]: Done uploading segments... Dec 27 04:54:56.771779040 (517h25m56.642862075s) [upload=453981]: Upload finished. err: <nil> Duration: 22.109120533s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.505086775s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:55:31 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.505112604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX] duration: 30.002919794s trace: https://jaeger.dev.storj.io/trace/69f0122fd61b9ab6 FAIL (2025-12-27 04:55:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ) context deadline exceeded ======LOGS====== Dec 27 04:54:53.228016161 (517h25m53.099099206s) [upload=453981]: Starting upload Dec 27 04:54:53.228085944 (517h25m53.099168989s) [upload=453981]: Done starting upload Dec 27 04:54:53.228222013 (517h25m53.099305058s) [upload=453981]: Uploading segments... Dec 27 04:54:53.228328165 (517h25m53.09941121s) [upload=453981]: Got next segment. Inline: false Dec 27 04:54:53.228364304 (517h25m53.099447349s) [upload=453981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:54:53.228369984 (517h25m53.099453029s) [upload=453981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:54:53.471176480 (517h25m53.342259515s) [seg_pos={0 0}, upload=453981]: Begin upload segment... Dec 27 04:54:53.471236604 (517h25m53.342319639s) [seg_pos={0 0}, upload=453981]: Done begin upload segment. Dec 27 04:54:53.471278964 (517h25m53.342361999s) [noise=true , address=145.79.227.10:4403 , piece_id=QJT2FXUJCWMPC5BEEU6INHKZEGCR2QGXLDW5YO55BPIIOU7REU3Q, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.471300586 (517h25m53.342383621s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.471799074 (517h25m53.34288211s) [noise=true , address=77.90.31.12:4590 , piece_id=W2JYHHHVVGT2D27ZYZA3HHKLO7Z5ATUAKJGR5Q5MPHVAZHGMAWWA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.472138040 (517h25m53.343221085s) [noise=true , address=145.79.227.2:4115 , piece_id=VHT5DS4Z5NJSNTNVGEERXQTAUKAUIHI7TOWPQBMZCN3QVFQQXSFQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473034377 (517h25m53.344117412s) [noise=true , address=145.79.227.45:4229 , piece_id=ILHC3BM7TDMHUS4ACIDFSZHPIYW7XPXELKJWP5CGO5AEE2737DYQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473402668 (517h25m53.344485703s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473761913 (517h25m53.344844948s) [noise=true , address=145.79.227.15:4874 , piece_id=R2MHZBZ63W47CYGDE3D7LPXSYLK4LTFJINUI3PJCBX3HKRIABK2A, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.474149651 (517h25m53.345232696s) [noise=true , address=45.91.116.5:4297 , piece_id=YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.475456570 (517h25m53.346539605s) [noise=true , address=77.90.31.21:4880 , piece_id=FVVU2G2JKJ6DTYHT62NAU7SM7M5C2NDERINEDXBTQH2PZTZAVVUQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.475590674 (517h25m53.346673719s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.476650784 (517h25m53.347733819s) [noise=true , address=77.90.31.11:4554 , piece_id=7A24GXKVWBS6KMNAQY5LEFQPS34FJVMUTSBWOCTAN4M24HHB3BCA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.477108996 (517h25m53.348192041s) [noise=true , address=145.79.227.11:4730 , piece_id=GAPDONNT4DIA5YW73JWWJKKNXLDC5UVIYD3IPMMV5OI3GRO3PBCA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.478711297 (517h25m53.349794332s) [noise=true , address=45.91.116.13:4154 , piece_id=5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.480251329 (517h25m53.351334374s) [noise=true , address=145.79.227.23:4546 , piece_id=HWOYTQHNDDXXSET3DLLUKPB45NWJJWHRQ2RQQRY47E2HLEJRQBLA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.481777846 (517h25m53.352860881s) [noise=true , address=145.79.227.49:4337 , piece_id=H5D75YGDK6SBGCLYHPN2GTQXPFT46SLUPYNJB3AMCRDF2V5NC4KA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.483148365 (517h25m53.3542314s) [noise=true , address=145.79.227.35:4483 , piece_id=5GWKPQ2SPRO6JUS5AH5K3OJMLVK4U5UQMPDKXUN5NPYZXJT656EA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:55.192400484 (517h25m55.063483519s) [upload=453981]: Next returned nil segment Dec 27 04:54:55.192418167 (517h25m55.063501202s) [upload=453981]: Waiting for error group managing segments... Dec 27 04:54:55.477444407 (517h25m55.348527452s) [noise=true , address=145.79.227.15:4874 , piece_id=R2MHZBZ63W47CYGDE3D7LPXSYLK4LTFJINUI3PJCBX3HKRIABK2A, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.480670369 (517h25m55.351753414s) [noise=true , address=145.79.227.10:4403 , piece_id=QJT2FXUJCWMPC5BEEU6INHKZEGCR2QGXLDW5YO55BPIIOU7REU3Q, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.481618815 (517h25m55.35270187s) [noise=true , address=145.79.227.23:4546 , piece_id=HWOYTQHNDDXXSET3DLLUKPB45NWJJWHRQ2RQQRY47E2HLEJRQBLA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.494611344 (517h25m55.365694389s) [noise=true , address=145.79.227.2:4115 , piece_id=VHT5DS4Z5NJSNTNVGEERXQTAUKAUIHI7TOWPQBMZCN3QVFQQXSFQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.499504542 (517h25m55.370587577s) [noise=true , address=145.79.227.35:4483 , piece_id=5GWKPQ2SPRO6JUS5AH5K3OJMLVK4U5UQMPDKXUN5NPYZXJT656EA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.504255118 (517h25m55.375338153s) [noise=true , address=145.79.227.45:4229 , piece_id=ILHC3BM7TDMHUS4ACIDFSZHPIYW7XPXELKJWP5CGO5AEE2737DYQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.508654526 (517h25m55.379737561s) [noise=true , address=145.79.227.11:4730 , piece_id=GAPDONNT4DIA5YW73JWWJKKNXLDC5UVIYD3IPMMV5OI3GRO3PBCA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.509571982 (517h25m55.380655027s) [noise=true , address=145.79.227.49:4337 , piece_id=H5D75YGDK6SBGCLYHPN2GTQXPFT46SLUPYNJB3AMCRDF2V5NC4KA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.664817878 (517h25m55.535900913s) [noise=true , address=77.90.31.12:4590 , piece_id=W2JYHHHVVGT2D27ZYZA3HHKLO7Z5ATUAKJGR5Q5MPHVAZHGMAWWA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.674947937 (517h25m55.546030982s) [noise=true , address=77.90.31.11:4554 , piece_id=7A24GXKVWBS6KMNAQY5LEFQPS34FJVMUTSBWOCTAN4M24HHB3BCA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.774217743 (517h25m55.645300788s) [noise=true , address=77.90.31.21:4880 , piece_id=FVVU2G2JKJ6DTYHT62NAU7SM7M5C2NDERINEDXBTQH2PZTZAVVUQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.836321479 (517h25m55.707404524s) [noise=true , address=45.91.116.5:4297 , piece_id=YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:56.479032291 (517h25m56.350115336s) [noise=true , address=45.91.116.13:4154 , piece_id=5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:56.479057088 (517h25m56.350140123s) [seg_pos={0 0}, upload=453981]: Segment reached optimal threshold of 13 pieces. Dec 27 04:54:56.479269112 (517h25m56.350352157s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479278970 (517h25m56.350362005s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479391665 (517h25m56.3504747s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479407575 (517h25m56.35049061s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479491635 (517h25m56.35057467s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479501274 (517h25m56.350584319s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479548243 (517h25m56.350631278s) [seg_pos={0 0}, upload=453981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:54:56.479628085 (517h25m56.35071113s) [upload=453981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:54:56.479633856 (517h25m56.350716901s) [upload=453981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:54:56.771755175 (517h25m56.642838211s) [upload=453981]: Done uploading segments... Dec 27 04:54:56.771779040 (517h25m56.642862075s) [upload=453981]: Upload finished. err: <nil> Duration: 30.002982605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X......................X.......] duration: 21.525440383s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:54:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 21.525468176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXX] duration: 30.004908993s trace: https://jaeger.dev.storj.io/trace/65cb8bf0e0677a4e FAIL (2025-12-27 04:54:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 04:54:13.459276285 (517h25m13.33035933s) [upload=453972]: Starting upload Dec 27 04:54:13.459398839 (517h25m13.330481884s) [upload=453972]: Done starting upload Dec 27 04:54:13.459437843 (517h25m13.330520888s) [upload=453972]: Uploading segments... Dec 27 04:54:13.459667781 (517h25m13.330750826s) [upload=453972]: Got next segment. Inline: false Dec 27 04:54:13.459779664 (517h25m13.330862709s) [upload=453972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:54:13.459787509 (517h25m13.330870554s) [upload=453972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:54:13.707210103 (517h25m13.578293148s) [seg_pos={0 0}, upload=453972]: Begin upload segment... Dec 27 04:54:13.707267613 (517h25m13.578350658s) [seg_pos={0 0}, upload=453972]: Done begin upload segment. Dec 27 04:54:13.707299774 (517h25m13.578382819s) [noise=true , address=145.79.227.42:4121 , piece_id=B3LBQRUVMFZXYGKGG7A326DDHNXDFQWTZDSP23JMHDJUPPUR6XMQ, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.709693082 (517h25m13.580776127s) [noise=true , address=77.90.31.21:4877 , piece_id=YM7ZY27JMUSAC5TD56MXL2RMELQXH5TFWDQRQ2XZDFMK2NXOYNLQ, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.710781184 (517h25m13.581864229s) [noise=true , address=145.79.227.17:4971 , piece_id=H2ZV7IOW26NMGUVL4IGZDT3XUC2J4DRFJJTO4Q3L57QZPVYDAN6A, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.711206854 (517h25m13.582289899s) [noise=true , address=145.79.227.50:4373 , piece_id=LISTLAKAVQSCYOZCSAYZD4SLFLT6BSWORWVAUJCREDYVAQX6YARA, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.711724299 (517h25m13.582807344s) [noise=true , address=145.79.227.19:4079 , piece_id=4Y5KSQHZZJUNE3LKJYZTVSJ6YBUSZHJ6UA33775PM2SR3YEIJYAQ, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.712211126 (517h25m13.583294171s) [noise=true , address=145.79.227.37:4565 , piece_id=ODGMFRI7YN3IWH6VOSYEU4LB65IIKJ36BNDWD2QC6IOCXE3NEFVQ, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.712876974 (517h25m13.583960019s) [noise=true , address=145.79.227.8:4331 , piece_id=4CPQMDVPLXASHRZZIYMOHY2VILIZOMGMLEF4VYRXTOUD7LJG6C4A, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.713697297 (517h25m13.584780342s) [noise=true , address=45.91.116.11:4117 , piece_id=6YCD2DNYNTA635WRXIIFB2LLG5EESWLRVHPNTSNXJASR7SBY3YIQ, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.714350621 (517h25m13.585433666s) [noise=true , address=45.91.116.12:4225 , piece_id=4VHSVPJ2H552TDKTRNWTY4P4X2ABRHQ3OPEHVB5TIDBDMD7KTAJA, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.716920664 (517h25m13.588003699s) [noise=true , address=145.79.227.3:4149 , piece_id=MDYJ5Z3KBPHI3GZQJ2RFOUPXJEYAK63HY3XRDUYEL2Q7BSD45ENA, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.718717726 (517h25m13.589800771s) [noise=true , address=77.90.31.14:4663 , piece_id=TL44OFMGZKTZWKO6VRCFPRLAUHMB5DLYRRL3PWJXFCHVT2B2LI3A, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.719971113 (517h25m13.591054148s) [noise=true , address=145.79.227.22:4511 , piece_id=Q7MYEUR662LGNRHDZUUIBWFDKYMURC2P3LHOZBXBPVCT4ZCNNEZQ, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.721254817 (517h25m13.592337862s) [noise=true , address=145.79.227.11:4732 , piece_id=Y5PA2YMLA6NULZU7UODCCJR6R3LUT5VSL72JC2TDPXISBETGR75Q, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.722740657 (517h25m13.593823702s) [noise=true , address=145.79.227.41:4085 , piece_id=FDRPTOR7Z6MRCDRLWRNSS52C7OFJ4GOG36NDZ2EEREHVKJHVGWTQ, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.724464840 (517h25m13.595547885s) [noise=true , address=45.91.116.6:4974 , piece_id=DRTD3H72N2UPLBKCZ3KLHXMYWBL42UE7BEZL6CI6WYUN4P3EON3A, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:13.725015629 (517h25m13.596098674s) [noise=true , address=77.90.31.11:4556 , piece_id=TCUEYTAG7Y3LSNHUN5H4TOC7XEVKUCU3DY7A32OXOTZVHFOLAPNQ, seg_pos={0 0}, upload=453972]: Uploading piece... Dec 27 04:54:15.435577110 (517h25m15.306660145s) [upload=453972]: Next returned nil segment Dec 27 04:54:15.435589313 (517h25m15.306672348s) [upload=453972]: Waiting for error group managing segments... Dec 27 04:54:15.721366250 (517h25m15.592449285s) [noise=true , address=145.79.227.8:4331 , piece_id=4CPQMDVPLXASHRZZIYMOHY2VILIZOMGMLEF4VYRXTOUD7LJG6C4A, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:15.721858639 (517h25m15.592941674s) [noise=true , address=145.79.227.37:4565 , piece_id=ODGMFRI7YN3IWH6VOSYEU4LB65IIKJ36BNDWD2QC6IOCXE3NEFVQ, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:15.722923055 (517h25m15.59400609s) [noise=true , address=145.79.227.3:4149 , piece_id=MDYJ5Z3KBPHI3GZQJ2RFOUPXJEYAK63HY3XRDUYEL2Q7BSD45ENA, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:15.724291251 (517h25m15.595374296s) [noise=true , address=145.79.227.17:4971 , piece_id=H2ZV7IOW26NMGUVL4IGZDT3XUC2J4DRFJJTO4Q3L57QZPVYDAN6A, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:15.726929534 (517h25m15.598012569s) [noise=true , address=145.79.227.41:4085 , piece_id=FDRPTOR7Z6MRCDRLWRNSS52C7OFJ4GOG36NDZ2EEREHVKJHVGWTQ, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:15.729581534 (517h25m15.600664569s) [noise=true , address=145.79.227.42:4121 , piece_id=B3LBQRUVMFZXYGKGG7A326DDHNXDFQWTZDSP23JMHDJUPPUR6XMQ, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:15.730083119 (517h25m15.601166154s) [noise=true , address=145.79.227.50:4373 , piece_id=LISTLAKAVQSCYOZCSAYZD4SLFLT6BSWORWVAUJCREDYVAQX6YARA, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:15.730746522 (517h25m15.601829567s) [noise=true , address=145.79.227.11:4732 , piece_id=Y5PA2YMLA6NULZU7UODCCJR6R3LUT5VSL72JC2TDPXISBETGR75Q, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:15.741535536 (517h25m15.612618572s) [noise=true , address=145.79.227.19:4079 , piece_id=4Y5KSQHZZJUNE3LKJYZTVSJ6YBUSZHJ6UA33775PM2SR3YEIJYAQ, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:15.754621474 (517h25m15.62570458s) [noise=true , address=145.79.227.22:4511 , piece_id=Q7MYEUR662LGNRHDZUUIBWFDKYMURC2P3LHOZBXBPVCT4ZCNNEZQ, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:15.926839267 (517h25m15.797922312s) [noise=true , address=77.90.31.14:4663 , piece_id=TL44OFMGZKTZWKO6VRCFPRLAUHMB5DLYRRL3PWJXFCHVT2B2LI3A, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:15.954701320 (517h25m15.825784355s) [noise=true , address=45.91.116.12:4225 , piece_id=4VHSVPJ2H552TDKTRNWTY4P4X2ABRHQ3OPEHVB5TIDBDMD7KTAJA, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:16.012588297 (517h25m15.883671332s) [noise=true , address=45.91.116.11:4117 , piece_id=6YCD2DNYNTA635WRXIIFB2LLG5EESWLRVHPNTSNXJASR7SBY3YIQ, seg_pos={0 0}, upload=453972]: Done uploading piece. err: <nil> Dec 27 04:54:16.012619346 (517h25m15.883702392s) [seg_pos={0 0}, upload=453972]: Segment reached optimal threshold of 13 pieces. Dec 27 04:54:16.012937683 (517h25m15.884020738s) [noise=true , address=77.90.31.21:4877 , piece_id=YM7ZY27JMUSAC5TD56MXL2RMELQXH5TFWDQRQ2XZDFMK2NXOYNLQ, seg_pos={0 0}, upload=453972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:16.012952190 (517h25m15.884035255s) [noise=true , address=77.90.31.21:4877 , piece_id=YM7ZY27JMUSAC5TD56MXL2RMELQXH5TFWDQRQ2XZDFMK2NXOYNLQ, seg_pos={0 0}, upload=453972]: Long tail cancellation due to optimal threshold Dec 27 04:54:16.013059895 (517h25m15.88414295s) [noise=true , address=77.90.31.11:4556 , piece_id=TCUEYTAG7Y3LSNHUN5H4TOC7XEVKUCU3DY7A32OXOTZVHFOLAPNQ, seg_pos={0 0}, upload=453972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:16.013081827 (517h25m15.884176043s) [noise=true , address=77.90.31.11:4556 , piece_id=TCUEYTAG7Y3LSNHUN5H4TOC7XEVKUCU3DY7A32OXOTZVHFOLAPNQ, seg_pos={0 0}, upload=453972]: Long tail cancellation due to optimal threshold Dec 27 04:54:16.013313388 (517h25m15.884396433s) [noise=true , address=45.91.116.6:4974 , piece_id=DRTD3H72N2UPLBKCZ3KLHXMYWBL42UE7BEZL6CI6WYUN4P3EON3A, seg_pos={0 0}, upload=453972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:44976->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:44976->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44976->45.91.116.6:4974: use of closed network connection Dec 27 04:54:16.013324158 (517h25m15.884407213s) [noise=true , address=45.91.116.6:4974 , piece_id=DRTD3H72N2UPLBKCZ3KLHXMYWBL42UE7BEZL6CI6WYUN4P3EON3A, seg_pos={0 0}, upload=453972]: Long tail cancellation due to optimal threshold Dec 27 04:54:16.013364305 (517h25m15.88444735s) [seg_pos={0 0}, upload=453972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:54:16.013502598 (517h25m15.884585653s) [upload=453972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:54:16.013511044 (517h25m15.884594109s) [upload=453972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:54:16.308690236 (517h25m16.179773281s) [upload=453972]: Done uploading segments... Dec 27 04:54:16.308710134 (517h25m16.179793179s) [upload=453972]: Upload finished. err: <nil> Duration: 30.004923469s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.533165931s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:54:30 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.533192032s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.454945356s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:54:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.454956547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XX] duration: 30.004420374s trace: https://jaeger.dev.storj.io/trace/665d4be34dbb4490 FAIL (2025-12-27 04:54:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 04:53:43.250710914 (517h24m43.121793949s) [upload=453966]: Starting upload Dec 27 04:53:43.250769726 (517h24m43.121852761s) [upload=453966]: Done starting upload Dec 27 04:53:43.250904723 (517h24m43.121987758s) [upload=453966]: Uploading segments... Dec 27 04:53:43.250992230 (517h24m43.122075275s) [upload=453966]: Got next segment. Inline: false Dec 27 04:53:43.251043908 (517h24m43.122126943s) [upload=453966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:53:43.251048847 (517h24m43.122131892s) [upload=453966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:53:43.497049524 (517h24m43.368132569s) [seg_pos={0 0}, upload=453966]: Begin upload segment... Dec 27 04:53:43.497126983 (517h24m43.368210028s) [seg_pos={0 0}, upload=453966]: Done begin upload segment. Dec 27 04:53:43.497307396 (517h24m43.368390461s) [noise=true , address=45.91.116.6:4974 , piece_id=OFO5EUGTYPL6JDNOBQPF3Q6PGPV73RD4ORKT3GFOF4N4V5SF5C6A, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.498123329 (517h24m43.369206364s) [noise=true , address=145.79.227.12:4765 , piece_id=SMC5BPEMD36M2NI35DGD2EZN3IZQDYZ7FURBU3BUUTACXWBPYL3A, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.498139410 (517h24m43.369222445s) [noise=true , address=77.90.31.15:4697 , piece_id=TYLVQOP3IDI6HUDMX6DPMGRBCE43I2ASFRA5GKPGQ2RCKISGT2DQ, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.498669580 (517h24m43.369752635s) [noise=true , address=145.79.227.48:4302 , piece_id=GIWKRR7BEJBTHBP7YM4GAWD2DKTVBMFDOQ3IBXYSOWIITIIHBXVQ, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.498691762 (517h24m43.369774807s) [noise=true , address=145.79.227.50:4373 , piece_id=JVAVH3UORRXSLOBXQ55B7DBNUISJLNWDUZT2DM42BZ4JC2FPLY4Q, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.499292346 (517h24m43.370375381s) [noise=true , address=145.79.227.29:4942 , piece_id=C33K2XTOJNMLSNBTAYAUZCDJUL35L6OM7SJ47LJYQKJSBYUMREUA, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.499304779 (517h24m43.370387824s) [noise=true , address=145.79.227.51:4409 , piece_id=G2UGZVY4CXRHNX4GMJH2TBJQJ4O2H7EBIUUHRMQYA5SVIOYDYYZA, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.499804120 (517h24m43.370887155s) [noise=true , address=145.79.227.11:4730 , piece_id=I2S77IUIZXPR5NT5U2JPZJQHYHDF66ZXTR6XSIJGDSMEG2NB24HA, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.499833195 (517h24m43.37091624s) [noise=true , address=45.91.116.8:4047 , piece_id=MAMW4YKXITGMARZQQUED3N2AZMYIL2VJ7JUDUYWBSJEO3DFB3MTQ, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.500417618 (517h24m43.371500663s) [noise=true , address=45.91.116.14:4191 , piece_id=J4RP6SVJM6A7NJ6BN22U43JZZ6RIMXLNX3I2I46KF2BFNXLXZVDQ, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.500495827 (517h24m43.371578872s) [noise=true , address=77.90.31.17:4770 , piece_id=MOBSBGQX2GIAJDR76KTAC6VBYJLFUFPXNX3EOZ7SJAZTWV2B3YAA, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.501034503 (517h24m43.372117538s) [noise=true , address=145.79.227.15:4874 , piece_id=FPGOGUEUCMQHGRCB3S76VGVXIRIA4YG25O7YEVK57RRHSDCJJALQ, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.501068227 (517h24m43.372151272s) [noise=true , address=145.79.227.6:4259 , piece_id=WQY4WUQBVKON2TB3PY7UXYPQS3N7TS4IHL2ZX7SR3K7ZYNZP2GDA, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.501727643 (517h24m43.372810678s) [noise=true , address=45.91.116.7:4011 , piece_id=WYFRINKAA75CYIZBO5QCA2Q3JVAX3JNKHJSAWSLXJNJAZC7OBAZQ, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.502751773 (517h24m43.373834828s) [noise=true , address=145.79.227.22:4511 , piece_id=JK3P3H4YTMNKC2NBM3XTDQC3AQ4RNMRGLUEYNWMG7ZP37CVWQYYA, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:43.502929801 (517h24m43.374012847s) [noise=true , address=145.79.227.36:4518 , piece_id=NQEUNI3MMQKFWO2W6SYTHN6DSHRTIJKAXA6COIV7VADW46JUTHAQ, seg_pos={0 0}, upload=453966]: Uploading piece... Dec 27 04:53:45.218627303 (517h24m45.089710348s) [upload=453966]: Next returned nil segment Dec 27 04:53:45.218643764 (517h24m45.089726809s) [upload=453966]: Waiting for error group managing segments... Dec 27 04:53:45.505180770 (517h24m45.376263815s) [noise=true , address=145.79.227.6:4259 , piece_id=WQY4WUQBVKON2TB3PY7UXYPQS3N7TS4IHL2ZX7SR3K7ZYNZP2GDA, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.506499281 (517h24m45.377582316s) [noise=true , address=145.79.227.50:4373 , piece_id=JVAVH3UORRXSLOBXQ55B7DBNUISJLNWDUZT2DM42BZ4JC2FPLY4Q, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.510538522 (517h24m45.381621567s) [noise=true , address=145.79.227.15:4874 , piece_id=FPGOGUEUCMQHGRCB3S76VGVXIRIA4YG25O7YEVK57RRHSDCJJALQ, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.513934188 (517h24m45.385017233s) [noise=true , address=145.79.227.22:4511 , piece_id=JK3P3H4YTMNKC2NBM3XTDQC3AQ4RNMRGLUEYNWMG7ZP37CVWQYYA, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.514771493 (517h24m45.385854528s) [noise=true , address=145.79.227.12:4765 , piece_id=SMC5BPEMD36M2NI35DGD2EZN3IZQDYZ7FURBU3BUUTACXWBPYL3A, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.515781215 (517h24m45.38686425s) [noise=true , address=145.79.227.29:4942 , piece_id=C33K2XTOJNMLSNBTAYAUZCDJUL35L6OM7SJ47LJYQKJSBYUMREUA, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.519650643 (517h24m45.390733688s) [noise=true , address=145.79.227.36:4518 , piece_id=NQEUNI3MMQKFWO2W6SYTHN6DSHRTIJKAXA6COIV7VADW46JUTHAQ, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.523019879 (517h24m45.394102924s) [noise=true , address=145.79.227.51:4409 , piece_id=G2UGZVY4CXRHNX4GMJH2TBJQJ4O2H7EBIUUHRMQYA5SVIOYDYYZA, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.524744322 (517h24m45.395827377s) [noise=true , address=145.79.227.48:4302 , piece_id=GIWKRR7BEJBTHBP7YM4GAWD2DKTVBMFDOQ3IBXYSOWIITIIHBXVQ, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.525821523 (517h24m45.396904568s) [noise=true , address=145.79.227.11:4730 , piece_id=I2S77IUIZXPR5NT5U2JPZJQHYHDF66ZXTR6XSIJGDSMEG2NB24HA, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.795636568 (517h24m45.666719603s) [noise=true , address=77.90.31.17:4770 , piece_id=MOBSBGQX2GIAJDR76KTAC6VBYJLFUFPXNX3EOZ7SJAZTWV2B3YAA, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.839396104 (517h24m45.710479139s) [noise=true , address=77.90.31.15:4697 , piece_id=TYLVQOP3IDI6HUDMX6DPMGRBCE43I2ASFRA5GKPGQ2RCKISGT2DQ, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.865953492 (517h24m45.737036538s) [noise=true , address=45.91.116.6:4974 , piece_id=OFO5EUGTYPL6JDNOBQPF3Q6PGPV73RD4ORKT3GFOF4N4V5SF5C6A, seg_pos={0 0}, upload=453966]: Done uploading piece. err: <nil> Dec 27 04:53:45.865974944 (517h24m45.737057989s) [seg_pos={0 0}, upload=453966]: Segment reached optimal threshold of 13 pieces. Dec 27 04:53:45.866197567 (517h24m45.737280602s) [noise=true , address=45.91.116.7:4011 , piece_id=WYFRINKAA75CYIZBO5QCA2Q3JVAX3JNKHJSAWSLXJNJAZC7OBAZQ, seg_pos={0 0}, upload=453966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 04:53:45.866204380 (517h24m45.737287415s) [noise=true , address=45.91.116.7:4011 , piece_id=WYFRINKAA75CYIZBO5QCA2Q3JVAX3JNKHJSAWSLXJNJAZC7OBAZQ, seg_pos={0 0}, upload=453966]: Long tail cancellation due to optimal threshold Dec 27 04:53:45.866511124 (517h24m45.737594169s) [noise=true , address=45.91.116.14:4191 , piece_id=J4RP6SVJM6A7NJ6BN22U43JZZ6RIMXLNX3I2I46KF2BFNXLXZVDQ, seg_pos={0 0}, upload=453966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 04:53:45.866519159 (517h24m45.737602194s) [noise=true , address=45.91.116.14:4191 , piece_id=J4RP6SVJM6A7NJ6BN22U43JZZ6RIMXLNX3I2I46KF2BFNXLXZVDQ, seg_pos={0 0}, upload=453966]: Long tail cancellation due to optimal threshold Dec 27 04:53:45.866629130 (517h24m45.737712175s) [noise=true , address=45.91.116.8:4047 , piece_id=MAMW4YKXITGMARZQQUED3N2AZMYIL2VJ7JUDUYWBSJEO3DFB3MTQ, seg_pos={0 0}, upload=453966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 04:53:45.866634920 (517h24m45.737717956s) [noise=true , address=45.91.116.8:4047 , piece_id=MAMW4YKXITGMARZQQUED3N2AZMYIL2VJ7JUDUYWBSJEO3DFB3MTQ, seg_pos={0 0}, upload=453966]: Long tail cancellation due to optimal threshold Dec 27 04:53:45.866649448 (517h24m45.737732483s) [seg_pos={0 0}, upload=453966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:53:45.866714963 (517h24m45.737798008s) [upload=453966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:53:45.866719471 (517h24m45.737802506s) [upload=453966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:53:46.162232218 (517h24m46.033315263s) [upload=453966]: Done uploading segments... Dec 27 04:53:46.162251254 (517h24m46.033334289s) [upload=453966]: Upload finished. err: <nil> Duration: 30.004538078s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X......................X......] duration: 45.783376894s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:54:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 45.783392804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....X] duration: 30.004521956s trace: https://jaeger.dev.storj.io/trace/33c81ec2b53163fe FAIL (2025-12-27 04:53:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 04:53:12.992560035 (517h24m12.863643081s) [upload=453959]: Starting upload Dec 27 04:53:12.992648012 (517h24m12.863731058s) [upload=453959]: Done starting upload Dec 27 04:53:12.992760386 (517h24m12.863843421s) [upload=453959]: Uploading segments... Dec 27 04:53:12.992935630 (517h24m12.864018675s) [upload=453959]: Got next segment. Inline: false Dec 27 04:53:12.992990495 (517h24m12.86407354s) [upload=453959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:53:12.992996987 (517h24m12.864080032s) [upload=453959]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:53:13.239738445 (517h24m13.11082149s) [seg_pos={0 0}, upload=453959]: Begin upload segment... Dec 27 04:53:13.239786618 (517h24m13.110869653s) [seg_pos={0 0}, upload=453959]: Done begin upload segment. Dec 27 04:53:13.239822566 (517h24m13.110905601s) [noise=true , address=145.79.227.48:4301 , piece_id=BSKRH4UON2ULHUNSX4U4EUXPJHU6TTKOSE572ZAAOKJQYDIKHKAQ, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.239829489 (517h24m13.110912534s) [noise=true , address=145.79.227.6:4259 , piece_id=ASS7VF6TJFJNP7MAHKLTVKXR4J4EDWNDX7GD6WFIE2YW2OJGPQHA, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.239876979 (517h24m13.110960024s) [noise=true , address=45.91.116.13:4156 , piece_id=2XXQUJWEUB6KVSMNC3ST5MNNJKPJ5EGXRPBY2N4MOSEPTEJAQGCA, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.240247294 (517h24m13.11133034s) [noise=true , address=77.90.31.19:4844 , piece_id=7GMIDOWAEUT5CERD52CT574TSDJXAZDPXGY3PHCPSNQPM2L4QJCQ, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.240568496 (517h24m13.111651541s) [noise=true , address=145.79.227.52:4446 , piece_id=DGPPG5LJ6E35SUQY2TJHWC3TXEQHJ3XTFOLEF664IJMHAG7ZL5WA, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.240962546 (517h24m13.112045591s) [noise=true , address=77.90.31.14:4664 , piece_id=XXAM2B6ZNNAK7NWDX2NK67567VTFWIZ4WC55IX2D52U2VAKDZCOA, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.241353000 (517h24m13.112436035s) [noise=true , address=145.79.227.20:4439 , piece_id=IAUNGIVAALGNCKXACL6YVABU32Q3HCSNYJPA6X3KOUC3CBYJ7HLQ, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.241701013 (517h24m13.112784048s) [noise=true , address=145.79.227.46:4603 , piece_id=C6WLKLZBZDZ7TKMZZWGRXWYY3WAUT4EGXX3JSW5NLNJTO5B4DIMA, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.243515277 (517h24m13.114598312s) [noise=true , address=77.90.31.10:4519 , piece_id=5RAQZ6TXAOUFRNFOMKT5CZ5QBEAHBQWEA77QBZNGQSA3XD5XZSQQ, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.244981780 (517h24m13.116064815s) [noise=true , address=145.79.227.12:4765 , piece_id=7KL5PVUUPZAMEOFGGQNYRG5UKBQCGW6AJSBPPJ35KTNJARW2OL5Q, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.246658272 (517h24m13.117741307s) [noise=true , address=145.79.227.39:4013 , piece_id=3JJTIZXNMUYTAKJLEEAHCDM4IEOXAMXRZ5GWQN4XIYW56R6KYNJA, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.247915716 (517h24m13.118998751s) [noise=true , address=145.79.227.11:4731 , piece_id=IOXBGRREMLQID4DGF7FVMTLGJQS4J4PVMHI47XOIMYYY3Q7VUDZA, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.248929166 (517h24m13.120012201s) [noise=true , address=45.91.116.14:4190 , piece_id=N7YC55U5O5CO3V624EICS2HIZXJ7D4LGPVADD673CDLCCYHFHJCQ, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.249774095 (517h24m13.120857131s) [noise=true , address=145.79.227.13:4804 , piece_id=GDGD3J5PMM2IS2RBJCKS3OVC4TB67CI4TFFA4WOWTN6OHLS3HOMA, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.250350452 (517h24m13.121433497s) [noise=true , address=77.90.31.12:4590 , piece_id=UTMFQWT2TOPFMQGRIXCSGIS5MBGWXLF2UUIMBZV6LLHY56E2LTUA, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:13.251473911 (517h24m13.122556956s) [noise=true , address=145.79.227.42:4121 , piece_id=ZP526FLK6PL4FRA6PBK4DMANCT4WWD6DRRKK7CHAW4I3SL2IYLOQ, seg_pos={0 0}, upload=453959]: Uploading piece... Dec 27 04:53:14.965845098 (517h24m14.836928133s) [upload=453959]: Next returned nil segment Dec 27 04:53:14.965865547 (517h24m14.836948642s) [upload=453959]: Waiting for error group managing segments... Dec 27 04:53:15.251020090 (517h24m15.122103125s) [noise=true , address=145.79.227.42:4121 , piece_id=ZP526FLK6PL4FRA6PBK4DMANCT4WWD6DRRKK7CHAW4I3SL2IYLOQ, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.251776330 (517h24m15.122859375s) [noise=true , address=145.79.227.20:4439 , piece_id=IAUNGIVAALGNCKXACL6YVABU32Q3HCSNYJPA6X3KOUC3CBYJ7HLQ, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.252926681 (517h24m15.124009726s) [noise=true , address=145.79.227.12:4765 , piece_id=7KL5PVUUPZAMEOFGGQNYRG5UKBQCGW6AJSBPPJ35KTNJARW2OL5Q, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.254954422 (517h24m15.126037467s) [noise=true , address=145.79.227.48:4301 , piece_id=BSKRH4UON2ULHUNSX4U4EUXPJHU6TTKOSE572ZAAOKJQYDIKHKAQ, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.258982883 (517h24m15.130065918s) [noise=true , address=145.79.227.13:4804 , piece_id=GDGD3J5PMM2IS2RBJCKS3OVC4TB67CI4TFFA4WOWTN6OHLS3HOMA, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.264150372 (517h24m15.135233407s) [noise=true , address=145.79.227.52:4446 , piece_id=DGPPG5LJ6E35SUQY2TJHWC3TXEQHJ3XTFOLEF664IJMHAG7ZL5WA, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.264811902 (517h24m15.135894937s) [noise=true , address=145.79.227.11:4731 , piece_id=IOXBGRREMLQID4DGF7FVMTLGJQS4J4PVMHI47XOIMYYY3Q7VUDZA, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.265207806 (517h24m15.136290841s) [noise=true , address=145.79.227.46:4603 , piece_id=C6WLKLZBZDZ7TKMZZWGRXWYY3WAUT4EGXX3JSW5NLNJTO5B4DIMA, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.276310127 (517h24m15.147393172s) [noise=true , address=145.79.227.39:4013 , piece_id=3JJTIZXNMUYTAKJLEEAHCDM4IEOXAMXRZ5GWQN4XIYW56R6KYNJA, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.287706507 (517h24m15.158789573s) [noise=true , address=145.79.227.6:4259 , piece_id=ASS7VF6TJFJNP7MAHKLTVKXR4J4EDWNDX7GD6WFIE2YW2OJGPQHA, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.430892628 (517h24m15.301975673s) [noise=true , address=77.90.31.12:4590 , piece_id=UTMFQWT2TOPFMQGRIXCSGIS5MBGWXLF2UUIMBZV6LLHY56E2LTUA, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.493258764 (517h24m15.364341829s) [noise=true , address=77.90.31.19:4844 , piece_id=7GMIDOWAEUT5CERD52CT574TSDJXAZDPXGY3PHCPSNQPM2L4QJCQ, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.502347931 (517h24m15.373430976s) [noise=true , address=77.90.31.14:4664 , piece_id=XXAM2B6ZNNAK7NWDX2NK67567VTFWIZ4WC55IX2D52U2VAKDZCOA, seg_pos={0 0}, upload=453959]: Done uploading piece. err: <nil> Dec 27 04:53:15.502374852 (517h24m15.373457887s) [seg_pos={0 0}, upload=453959]: Segment reached optimal threshold of 13 pieces. Dec 27 04:53:15.502670475 (517h24m15.37375351s) [noise=true , address=45.91.116.14:4190 , piece_id=N7YC55U5O5CO3V624EICS2HIZXJ7D4LGPVADD673CDLCCYHFHJCQ, seg_pos={0 0}, upload=453959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 04:53:15.502682899 (517h24m15.373765944s) [noise=true , address=45.91.116.14:4190 , piece_id=N7YC55U5O5CO3V624EICS2HIZXJ7D4LGPVADD673CDLCCYHFHJCQ, seg_pos={0 0}, upload=453959]: Long tail cancellation due to optimal threshold Dec 27 04:53:15.502995114 (517h24m15.374078169s) [noise=true , address=45.91.116.13:4156 , piece_id=2XXQUJWEUB6KVSMNC3ST5MNNJKPJ5EGXRPBY2N4MOSEPTEJAQGCA, seg_pos={0 0}, upload=453959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 04:53:15.503004050 (517h24m15.374087085s) [noise=true , address=45.91.116.13:4156 , piece_id=2XXQUJWEUB6KVSMNC3ST5MNNJKPJ5EGXRPBY2N4MOSEPTEJAQGCA, seg_pos={0 0}, upload=453959]: Long tail cancellation due to optimal threshold Dec 27 04:53:15.503160037 (517h24m15.374243092s) [noise=true , address=77.90.31.10:4519 , piece_id=5RAQZ6TXAOUFRNFOMKT5CZ5QBEAHBQWEA77QBZNGQSA3XD5XZSQQ, seg_pos={0 0}, upload=453959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 27 04:53:15.503168053 (517h24m15.374251098s) [noise=true , address=77.90.31.10:4519 , piece_id=5RAQZ6TXAOUFRNFOMKT5CZ5QBEAHBQWEA77QBZNGQSA3XD5XZSQQ, seg_pos={0 0}, upload=453959]: Long tail cancellation due to optimal threshold Dec 27 04:53:15.503194212 (517h24m15.374277247s) [seg_pos={0 0}, upload=453959]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:53:15.503279585 (517h24m15.37436262s) [upload=453959]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:53:15.503285727 (517h24m15.374368772s) [upload=453959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:53:15.800387399 (517h24m15.671470434s) [upload=453959]: Done uploading segments... Dec 27 04:53:15.800414851 (517h24m15.671497896s) [upload=453959]: Upload finished. err: <nil> Duration: 30.004532465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X......................X.....] duration: 43.312618369s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:53:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 43.312642565s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.57030538s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:53:17 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.570349145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.XX..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....] duration: 27.243921574s trace: https://jaeger.dev.storj.io/trace/5e90b03efa41de35 FAIL (2025-12-27 04:53:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 04:50:44.473653280 (517h21m44.344736325s) [upload=453922]: Starting upload Dec 27 04:50:44.473725197 (517h21m44.344808242s) [upload=453922]: Done starting upload Dec 27 04:50:44.473773379 (517h21m44.344856424s) [upload=453922]: Uploading segments... Dec 27 04:50:44.473930388 (517h21m44.345013423s) [upload=453922]: Got next segment. Inline: false Dec 27 04:50:44.473966126 (517h21m44.345049161s) [upload=453922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:50:44.473970133 (517h21m44.345053168s) [upload=453922]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:50:44.720782557 (517h21m44.591865592s) [seg_pos={0 0}, upload=453922]: Begin upload segment... Dec 27 04:50:44.720855896 (517h21m44.591938931s) [seg_pos={0 0}, upload=453922]: Done begin upload segment. Dec 27 04:50:44.720939585 (517h21m44.59202263s) [noise=true , address=77.90.31.11:4554 , piece_id=2XBKJBOIIVL5YSTWPJBGKDWOM6K5ZVOAJT7AQZVKJGFPFQEKEQMQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.720989631 (517h21m44.592072666s) [noise=true , address=145.79.227.4:4185 , piece_id=EHX3JCSZHGKQ3RS5DAH7LINNJO6ERSVN7J7JUHXFSW2HFO546ZXA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.721643657 (517h21m44.592726692s) [noise=true , address=45.91.116.6:4974 , piece_id=TVLEEJK6ENP326EF4B6PERDX43NSZ4GFT4OECQ3J55OUCZ7ZY6DQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.722233099 (517h21m44.593316134s) [noise=true , address=145.79.227.39:4014 , piece_id=5ZIKJPTBAI33C4GCEHMZJWZRA3DWUAYR3YKHEDFFABC5CJDXBJWQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.722786612 (517h21m44.593869647s) [noise=true , address=145.79.227.17:4972 , piece_id=3HQMNCQJFBXFB3WA2TKY7F4R7EVF7O3WYSE6RZCRHEAG4VOJ23XQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.723175383 (517h21m44.594258418s) [noise=true , address=77.90.31.21:4880 , piece_id=T22IJQPEDWDCM23NOYUMDMPOQIXGMTVJEOGMSILDZKVK72VWNI3A, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.723600533 (517h21m44.594683588s) [noise=true , address=45.91.116.12:4228 , piece_id=I6QUV2ODXGGCREOKKBKF6XHKSCQIO64ETYUJ4ZBNEQSMQRQB6QGA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.724048866 (517h21m44.595131901s) [noise=true , address=145.79.227.35:4483 , piece_id=XN52OMDOQXQNUNZGZE4RLEN3NPOCJVRPWT2BMF5JNCD6AV4W24YQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.725054301 (517h21m44.596137346s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.725518304 (517h21m44.596601349s) [noise=true , address=145.79.227.2:4114 , piece_id=T7RSPUEOXEABLHS357OMDDY5S47L7Y2VBURFLNM4BEK42F4SKUEQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.726666771 (517h21m44.597749816s) [noise=true , address=145.79.227.38:4978 , piece_id=3ALOI3LKLPJINDAPR4UL6STPPZFUCQN5OMZAHMWDIXGOHYSQ54BQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.728073139 (517h21m44.599156174s) [noise=true , address=145.79.227.52:4447 , piece_id=MCDADETIJYZVDDELUE3OV7K7FYT6M7WTE6PEEJV2PKJSA2UYKCRA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.729538148 (517h21m44.600621193s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.730874213 (517h21m44.601957258s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.733746442 (517h21m44.604829477s) [noise=true , address=145.79.227.26:4654 , piece_id=5L4VCMGH6ZPNMMAACNCLDHJ5ACGTWD7BCFC6R4GIJXALUQOGT5EA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.736813432 (517h21m44.607896467s) [noise=true , address=145.79.227.36:4518 , piece_id=DQ6VYB4X5J6EXPXAGSRAYXMOFGHZJBFJJIINP5WMJNBOLP4HWLCQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:46.349325320 (517h21m46.220408355s) [upload=453922]: Next returned nil segment Dec 27 04:50:46.349563915 (517h21m46.22064695s) [upload=453922]: Waiting for error group managing segments... Dec 27 04:50:46.728761261 (517h21m46.599844306s) [noise=true , address=145.79.227.36:4518 , piece_id=DQ6VYB4X5J6EXPXAGSRAYXMOFGHZJBFJJIINP5WMJNBOLP4HWLCQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.735006383 (517h21m46.606089418s) [noise=true , address=145.79.227.35:4483 , piece_id=XN52OMDOQXQNUNZGZE4RLEN3NPOCJVRPWT2BMF5JNCD6AV4W24YQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.736280198 (517h21m46.607363233s) [noise=true , address=145.79.227.39:4014 , piece_id=5ZIKJPTBAI33C4GCEHMZJWZRA3DWUAYR3YKHEDFFABC5CJDXBJWQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.736925367 (517h21m46.608008412s) [noise=true , address=145.79.227.38:4978 , piece_id=3ALOI3LKLPJINDAPR4UL6STPPZFUCQN5OMZAHMWDIXGOHYSQ54BQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.744323314 (517h21m46.615406349s) [noise=true , address=145.79.227.26:4654 , piece_id=5L4VCMGH6ZPNMMAACNCLDHJ5ACGTWD7BCFC6R4GIJXALUQOGT5EA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.748123350 (517h21m46.619206416s) [noise=true , address=145.79.227.17:4972 , piece_id=3HQMNCQJFBXFB3WA2TKY7F4R7EVF7O3WYSE6RZCRHEAG4VOJ23XQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.759159916 (517h21m46.630242961s) [noise=true , address=145.79.227.2:4114 , piece_id=T7RSPUEOXEABLHS357OMDDY5S47L7Y2VBURFLNM4BEK42F4SKUEQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.762073694 (517h21m46.633156739s) [noise=true , address=145.79.227.4:4185 , piece_id=EHX3JCSZHGKQ3RS5DAH7LINNJO6ERSVN7J7JUHXFSW2HFO546ZXA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.774638660 (517h21m46.645721705s) [noise=true , address=145.79.227.52:4447 , piece_id=MCDADETIJYZVDDELUE3OV7K7FYT6M7WTE6PEEJV2PKJSA2UYKCRA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.989846352 (517h21m46.860929397s) [noise=true , address=45.91.116.6:4974 , piece_id=TVLEEJK6ENP326EF4B6PERDX43NSZ4GFT4OECQ3J55OUCZ7ZY6DQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.052153085 (517h21m46.92323613s) [noise=true , address=77.90.31.21:4880 , piece_id=T22IJQPEDWDCM23NOYUMDMPOQIXGMTVJEOGMSILDZKVK72VWNI3A, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.125333994 (517h21m46.996417029s) [noise=true , address=77.90.31.11:4554 , piece_id=2XBKJBOIIVL5YSTWPJBGKDWOM6K5ZVOAJT7AQZVKJGFPFQEKEQMQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.129239290 (517h21m47.000322335s) [noise=true , address=45.91.116.12:4228 , piece_id=I6QUV2ODXGGCREOKKBKF6XHKSCQIO64ETYUJ4ZBNEQSMQRQB6QGA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.129256633 (517h21m47.000339668s) [seg_pos={0 0}, upload=453922]: Segment reached optimal threshold of 13 pieces. Dec 27 04:50:47.129433039 (517h21m47.000516104s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129442657 (517h21m47.000525702s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129819515 (517h21m47.00090255s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129826208 (517h21m47.000909253s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129954863 (517h21m47.001037898s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129962427 (517h21m47.001045462s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129983497 (517h21m47.001066532s) [seg_pos={0 0}, upload=453922]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:50:47.130050715 (517h21m47.00113375s) [upload=453922]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:50:47.130056877 (517h21m47.001139922s) [upload=453922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:50:47.421324630 (517h21m47.292407675s) [upload=453922]: Done uploading segments... Dec 27 04:50:47.421344989 (517h21m47.292428024s) [upload=453922]: Upload finished. err: <nil> Duration: 27.243950939s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X......................X....] duration: 51.671137445s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:52:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 51.671151923s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.544291455s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:52:21 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.544328115s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.509530993s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:52:05 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.509572974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.X.XX..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX...] duration: 6.219613306s trace: https://jaeger.dev.storj.io/trace/5e90b03efa41de35 FAIL (2025-12-27 04:52:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 04:50:44.473653280 (517h21m44.344736325s) [upload=453922]: Starting upload Dec 27 04:50:44.473725197 (517h21m44.344808242s) [upload=453922]: Done starting upload Dec 27 04:50:44.473773379 (517h21m44.344856424s) [upload=453922]: Uploading segments... Dec 27 04:50:44.473930388 (517h21m44.345013423s) [upload=453922]: Got next segment. Inline: false Dec 27 04:50:44.473966126 (517h21m44.345049161s) [upload=453922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:50:44.473970133 (517h21m44.345053168s) [upload=453922]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:50:44.720782557 (517h21m44.591865592s) [seg_pos={0 0}, upload=453922]: Begin upload segment... Dec 27 04:50:44.720855896 (517h21m44.591938931s) [seg_pos={0 0}, upload=453922]: Done begin upload segment. Dec 27 04:50:44.720939585 (517h21m44.59202263s) [noise=true , address=77.90.31.11:4554 , piece_id=2XBKJBOIIVL5YSTWPJBGKDWOM6K5ZVOAJT7AQZVKJGFPFQEKEQMQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.720989631 (517h21m44.592072666s) [noise=true , address=145.79.227.4:4185 , piece_id=EHX3JCSZHGKQ3RS5DAH7LINNJO6ERSVN7J7JUHXFSW2HFO546ZXA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.721643657 (517h21m44.592726692s) [noise=true , address=45.91.116.6:4974 , piece_id=TVLEEJK6ENP326EF4B6PERDX43NSZ4GFT4OECQ3J55OUCZ7ZY6DQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.722233099 (517h21m44.593316134s) [noise=true , address=145.79.227.39:4014 , piece_id=5ZIKJPTBAI33C4GCEHMZJWZRA3DWUAYR3YKHEDFFABC5CJDXBJWQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.722786612 (517h21m44.593869647s) [noise=true , address=145.79.227.17:4972 , piece_id=3HQMNCQJFBXFB3WA2TKY7F4R7EVF7O3WYSE6RZCRHEAG4VOJ23XQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.723175383 (517h21m44.594258418s) [noise=true , address=77.90.31.21:4880 , piece_id=T22IJQPEDWDCM23NOYUMDMPOQIXGMTVJEOGMSILDZKVK72VWNI3A, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.723600533 (517h21m44.594683588s) [noise=true , address=45.91.116.12:4228 , piece_id=I6QUV2ODXGGCREOKKBKF6XHKSCQIO64ETYUJ4ZBNEQSMQRQB6QGA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.724048866 (517h21m44.595131901s) [noise=true , address=145.79.227.35:4483 , piece_id=XN52OMDOQXQNUNZGZE4RLEN3NPOCJVRPWT2BMF5JNCD6AV4W24YQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.725054301 (517h21m44.596137346s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.725518304 (517h21m44.596601349s) [noise=true , address=145.79.227.2:4114 , piece_id=T7RSPUEOXEABLHS357OMDDY5S47L7Y2VBURFLNM4BEK42F4SKUEQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.726666771 (517h21m44.597749816s) [noise=true , address=145.79.227.38:4978 , piece_id=3ALOI3LKLPJINDAPR4UL6STPPZFUCQN5OMZAHMWDIXGOHYSQ54BQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.728073139 (517h21m44.599156174s) [noise=true , address=145.79.227.52:4447 , piece_id=MCDADETIJYZVDDELUE3OV7K7FYT6M7WTE6PEEJV2PKJSA2UYKCRA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.729538148 (517h21m44.600621193s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.730874213 (517h21m44.601957258s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.733746442 (517h21m44.604829477s) [noise=true , address=145.79.227.26:4654 , piece_id=5L4VCMGH6ZPNMMAACNCLDHJ5ACGTWD7BCFC6R4GIJXALUQOGT5EA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.736813432 (517h21m44.607896467s) [noise=true , address=145.79.227.36:4518 , piece_id=DQ6VYB4X5J6EXPXAGSRAYXMOFGHZJBFJJIINP5WMJNBOLP4HWLCQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:46.349325320 (517h21m46.220408355s) [upload=453922]: Next returned nil segment Dec 27 04:50:46.349563915 (517h21m46.22064695s) [upload=453922]: Waiting for error group managing segments... Dec 27 04:50:46.728761261 (517h21m46.599844306s) [noise=true , address=145.79.227.36:4518 , piece_id=DQ6VYB4X5J6EXPXAGSRAYXMOFGHZJBFJJIINP5WMJNBOLP4HWLCQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.735006383 (517h21m46.606089418s) [noise=true , address=145.79.227.35:4483 , piece_id=XN52OMDOQXQNUNZGZE4RLEN3NPOCJVRPWT2BMF5JNCD6AV4W24YQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.736280198 (517h21m46.607363233s) [noise=true , address=145.79.227.39:4014 , piece_id=5ZIKJPTBAI33C4GCEHMZJWZRA3DWUAYR3YKHEDFFABC5CJDXBJWQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.736925367 (517h21m46.608008412s) [noise=true , address=145.79.227.38:4978 , piece_id=3ALOI3LKLPJINDAPR4UL6STPPZFUCQN5OMZAHMWDIXGOHYSQ54BQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.744323314 (517h21m46.615406349s) [noise=true , address=145.79.227.26:4654 , piece_id=5L4VCMGH6ZPNMMAACNCLDHJ5ACGTWD7BCFC6R4GIJXALUQOGT5EA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.748123350 (517h21m46.619206416s) [noise=true , address=145.79.227.17:4972 , piece_id=3HQMNCQJFBXFB3WA2TKY7F4R7EVF7O3WYSE6RZCRHEAG4VOJ23XQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.759159916 (517h21m46.630242961s) [noise=true , address=145.79.227.2:4114 , piece_id=T7RSPUEOXEABLHS357OMDDY5S47L7Y2VBURFLNM4BEK42F4SKUEQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.762073694 (517h21m46.633156739s) [noise=true , address=145.79.227.4:4185 , piece_id=EHX3JCSZHGKQ3RS5DAH7LINNJO6ERSVN7J7JUHXFSW2HFO546ZXA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.774638660 (517h21m46.645721705s) [noise=true , address=145.79.227.52:4447 , piece_id=MCDADETIJYZVDDELUE3OV7K7FYT6M7WTE6PEEJV2PKJSA2UYKCRA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.989846352 (517h21m46.860929397s) [noise=true , address=45.91.116.6:4974 , piece_id=TVLEEJK6ENP326EF4B6PERDX43NSZ4GFT4OECQ3J55OUCZ7ZY6DQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.052153085 (517h21m46.92323613s) [noise=true , address=77.90.31.21:4880 , piece_id=T22IJQPEDWDCM23NOYUMDMPOQIXGMTVJEOGMSILDZKVK72VWNI3A, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.125333994 (517h21m46.996417029s) [noise=true , address=77.90.31.11:4554 , piece_id=2XBKJBOIIVL5YSTWPJBGKDWOM6K5ZVOAJT7AQZVKJGFPFQEKEQMQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.129239290 (517h21m47.000322335s) [noise=true , address=45.91.116.12:4228 , piece_id=I6QUV2ODXGGCREOKKBKF6XHKSCQIO64ETYUJ4ZBNEQSMQRQB6QGA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.129256633 (517h21m47.000339668s) [seg_pos={0 0}, upload=453922]: Segment reached optimal threshold of 13 pieces. Dec 27 04:50:47.129433039 (517h21m47.000516104s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129442657 (517h21m47.000525702s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129819515 (517h21m47.00090255s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129826208 (517h21m47.000909253s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129954863 (517h21m47.001037898s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129962427 (517h21m47.001045462s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129983497 (517h21m47.001066532s) [seg_pos={0 0}, upload=453922]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:50:47.130050715 (517h21m47.00113375s) [upload=453922]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:50:47.130056877 (517h21m47.001139922s) [upload=453922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:50:47.421324630 (517h21m47.292407675s) [upload=453922]: Done uploading segments... Dec 27 04:50:47.421344989 (517h21m47.292428024s) [upload=453922]: Upload finished. err: <nil> Duration: 6.219676146s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X......................X...] duration: 43.37801057s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:51:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 43.378039786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..X.XX..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX..] duration: 5.844726769s trace: https://jaeger.dev.storj.io/trace/5e90b03efa41de35 FAIL (2025-12-27 04:51:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 04:50:44.473653280 (517h21m44.344736325s) [upload=453922]: Starting upload Dec 27 04:50:44.473725197 (517h21m44.344808242s) [upload=453922]: Done starting upload Dec 27 04:50:44.473773379 (517h21m44.344856424s) [upload=453922]: Uploading segments... Dec 27 04:50:44.473930388 (517h21m44.345013423s) [upload=453922]: Got next segment. Inline: false Dec 27 04:50:44.473966126 (517h21m44.345049161s) [upload=453922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:50:44.473970133 (517h21m44.345053168s) [upload=453922]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:50:44.720782557 (517h21m44.591865592s) [seg_pos={0 0}, upload=453922]: Begin upload segment... Dec 27 04:50:44.720855896 (517h21m44.591938931s) [seg_pos={0 0}, upload=453922]: Done begin upload segment. Dec 27 04:50:44.720939585 (517h21m44.59202263s) [noise=true , address=77.90.31.11:4554 , piece_id=2XBKJBOIIVL5YSTWPJBGKDWOM6K5ZVOAJT7AQZVKJGFPFQEKEQMQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.720989631 (517h21m44.592072666s) [noise=true , address=145.79.227.4:4185 , piece_id=EHX3JCSZHGKQ3RS5DAH7LINNJO6ERSVN7J7JUHXFSW2HFO546ZXA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.721643657 (517h21m44.592726692s) [noise=true , address=45.91.116.6:4974 , piece_id=TVLEEJK6ENP326EF4B6PERDX43NSZ4GFT4OECQ3J55OUCZ7ZY6DQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.722233099 (517h21m44.593316134s) [noise=true , address=145.79.227.39:4014 , piece_id=5ZIKJPTBAI33C4GCEHMZJWZRA3DWUAYR3YKHEDFFABC5CJDXBJWQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.722786612 (517h21m44.593869647s) [noise=true , address=145.79.227.17:4972 , piece_id=3HQMNCQJFBXFB3WA2TKY7F4R7EVF7O3WYSE6RZCRHEAG4VOJ23XQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.723175383 (517h21m44.594258418s) [noise=true , address=77.90.31.21:4880 , piece_id=T22IJQPEDWDCM23NOYUMDMPOQIXGMTVJEOGMSILDZKVK72VWNI3A, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.723600533 (517h21m44.594683588s) [noise=true , address=45.91.116.12:4228 , piece_id=I6QUV2ODXGGCREOKKBKF6XHKSCQIO64ETYUJ4ZBNEQSMQRQB6QGA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.724048866 (517h21m44.595131901s) [noise=true , address=145.79.227.35:4483 , piece_id=XN52OMDOQXQNUNZGZE4RLEN3NPOCJVRPWT2BMF5JNCD6AV4W24YQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.725054301 (517h21m44.596137346s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.725518304 (517h21m44.596601349s) [noise=true , address=145.79.227.2:4114 , piece_id=T7RSPUEOXEABLHS357OMDDY5S47L7Y2VBURFLNM4BEK42F4SKUEQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.726666771 (517h21m44.597749816s) [noise=true , address=145.79.227.38:4978 , piece_id=3ALOI3LKLPJINDAPR4UL6STPPZFUCQN5OMZAHMWDIXGOHYSQ54BQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.728073139 (517h21m44.599156174s) [noise=true , address=145.79.227.52:4447 , piece_id=MCDADETIJYZVDDELUE3OV7K7FYT6M7WTE6PEEJV2PKJSA2UYKCRA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.729538148 (517h21m44.600621193s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.730874213 (517h21m44.601957258s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.733746442 (517h21m44.604829477s) [noise=true , address=145.79.227.26:4654 , piece_id=5L4VCMGH6ZPNMMAACNCLDHJ5ACGTWD7BCFC6R4GIJXALUQOGT5EA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.736813432 (517h21m44.607896467s) [noise=true , address=145.79.227.36:4518 , piece_id=DQ6VYB4X5J6EXPXAGSRAYXMOFGHZJBFJJIINP5WMJNBOLP4HWLCQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:46.349325320 (517h21m46.220408355s) [upload=453922]: Next returned nil segment Dec 27 04:50:46.349563915 (517h21m46.22064695s) [upload=453922]: Waiting for error group managing segments... Dec 27 04:50:46.728761261 (517h21m46.599844306s) [noise=true , address=145.79.227.36:4518 , piece_id=DQ6VYB4X5J6EXPXAGSRAYXMOFGHZJBFJJIINP5WMJNBOLP4HWLCQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.735006383 (517h21m46.606089418s) [noise=true , address=145.79.227.35:4483 , piece_id=XN52OMDOQXQNUNZGZE4RLEN3NPOCJVRPWT2BMF5JNCD6AV4W24YQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.736280198 (517h21m46.607363233s) [noise=true , address=145.79.227.39:4014 , piece_id=5ZIKJPTBAI33C4GCEHMZJWZRA3DWUAYR3YKHEDFFABC5CJDXBJWQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.736925367 (517h21m46.608008412s) [noise=true , address=145.79.227.38:4978 , piece_id=3ALOI3LKLPJINDAPR4UL6STPPZFUCQN5OMZAHMWDIXGOHYSQ54BQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.744323314 (517h21m46.615406349s) [noise=true , address=145.79.227.26:4654 , piece_id=5L4VCMGH6ZPNMMAACNCLDHJ5ACGTWD7BCFC6R4GIJXALUQOGT5EA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.748123350 (517h21m46.619206416s) [noise=true , address=145.79.227.17:4972 , piece_id=3HQMNCQJFBXFB3WA2TKY7F4R7EVF7O3WYSE6RZCRHEAG4VOJ23XQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.759159916 (517h21m46.630242961s) [noise=true , address=145.79.227.2:4114 , piece_id=T7RSPUEOXEABLHS357OMDDY5S47L7Y2VBURFLNM4BEK42F4SKUEQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.762073694 (517h21m46.633156739s) [noise=true , address=145.79.227.4:4185 , piece_id=EHX3JCSZHGKQ3RS5DAH7LINNJO6ERSVN7J7JUHXFSW2HFO546ZXA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.774638660 (517h21m46.645721705s) [noise=true , address=145.79.227.52:4447 , piece_id=MCDADETIJYZVDDELUE3OV7K7FYT6M7WTE6PEEJV2PKJSA2UYKCRA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.989846352 (517h21m46.860929397s) [noise=true , address=45.91.116.6:4974 , piece_id=TVLEEJK6ENP326EF4B6PERDX43NSZ4GFT4OECQ3J55OUCZ7ZY6DQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.052153085 (517h21m46.92323613s) [noise=true , address=77.90.31.21:4880 , piece_id=T22IJQPEDWDCM23NOYUMDMPOQIXGMTVJEOGMSILDZKVK72VWNI3A, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.125333994 (517h21m46.996417029s) [noise=true , address=77.90.31.11:4554 , piece_id=2XBKJBOIIVL5YSTWPJBGKDWOM6K5ZVOAJT7AQZVKJGFPFQEKEQMQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.129239290 (517h21m47.000322335s) [noise=true , address=45.91.116.12:4228 , piece_id=I6QUV2ODXGGCREOKKBKF6XHKSCQIO64ETYUJ4ZBNEQSMQRQB6QGA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.129256633 (517h21m47.000339668s) [seg_pos={0 0}, upload=453922]: Segment reached optimal threshold of 13 pieces. Dec 27 04:50:47.129433039 (517h21m47.000516104s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129442657 (517h21m47.000525702s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129819515 (517h21m47.00090255s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129826208 (517h21m47.000909253s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129954863 (517h21m47.001037898s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129962427 (517h21m47.001045462s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129983497 (517h21m47.001066532s) [seg_pos={0 0}, upload=453922]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:50:47.130050715 (517h21m47.00113375s) [upload=453922]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:50:47.130056877 (517h21m47.001139922s) [upload=453922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:50:47.421324630 (517h21m47.292407675s) [upload=453922]: Done uploading segments... Dec 27 04:50:47.421344989 (517h21m47.292428024s) [upload=453922]: Upload finished. err: <nil> Duration: 5.844754471s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [...X.XX..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX.] duration: 13.465655806s trace: https://jaeger.dev.storj.io/trace/5e90b03efa41de35 FAIL (2025-12-27 04:51:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 04:50:44.473653280 (517h21m44.344736325s) [upload=453922]: Starting upload Dec 27 04:50:44.473725197 (517h21m44.344808242s) [upload=453922]: Done starting upload Dec 27 04:50:44.473773379 (517h21m44.344856424s) [upload=453922]: Uploading segments... Dec 27 04:50:44.473930388 (517h21m44.345013423s) [upload=453922]: Got next segment. Inline: false Dec 27 04:50:44.473966126 (517h21m44.345049161s) [upload=453922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:50:44.473970133 (517h21m44.345053168s) [upload=453922]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:50:44.720782557 (517h21m44.591865592s) [seg_pos={0 0}, upload=453922]: Begin upload segment... Dec 27 04:50:44.720855896 (517h21m44.591938931s) [seg_pos={0 0}, upload=453922]: Done begin upload segment. Dec 27 04:50:44.720939585 (517h21m44.59202263s) [noise=true , address=77.90.31.11:4554 , piece_id=2XBKJBOIIVL5YSTWPJBGKDWOM6K5ZVOAJT7AQZVKJGFPFQEKEQMQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.720989631 (517h21m44.592072666s) [noise=true , address=145.79.227.4:4185 , piece_id=EHX3JCSZHGKQ3RS5DAH7LINNJO6ERSVN7J7JUHXFSW2HFO546ZXA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.721643657 (517h21m44.592726692s) [noise=true , address=45.91.116.6:4974 , piece_id=TVLEEJK6ENP326EF4B6PERDX43NSZ4GFT4OECQ3J55OUCZ7ZY6DQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.722233099 (517h21m44.593316134s) [noise=true , address=145.79.227.39:4014 , piece_id=5ZIKJPTBAI33C4GCEHMZJWZRA3DWUAYR3YKHEDFFABC5CJDXBJWQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.722786612 (517h21m44.593869647s) [noise=true , address=145.79.227.17:4972 , piece_id=3HQMNCQJFBXFB3WA2TKY7F4R7EVF7O3WYSE6RZCRHEAG4VOJ23XQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.723175383 (517h21m44.594258418s) [noise=true , address=77.90.31.21:4880 , piece_id=T22IJQPEDWDCM23NOYUMDMPOQIXGMTVJEOGMSILDZKVK72VWNI3A, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.723600533 (517h21m44.594683588s) [noise=true , address=45.91.116.12:4228 , piece_id=I6QUV2ODXGGCREOKKBKF6XHKSCQIO64ETYUJ4ZBNEQSMQRQB6QGA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.724048866 (517h21m44.595131901s) [noise=true , address=145.79.227.35:4483 , piece_id=XN52OMDOQXQNUNZGZE4RLEN3NPOCJVRPWT2BMF5JNCD6AV4W24YQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.725054301 (517h21m44.596137346s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.725518304 (517h21m44.596601349s) [noise=true , address=145.79.227.2:4114 , piece_id=T7RSPUEOXEABLHS357OMDDY5S47L7Y2VBURFLNM4BEK42F4SKUEQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.726666771 (517h21m44.597749816s) [noise=true , address=145.79.227.38:4978 , piece_id=3ALOI3LKLPJINDAPR4UL6STPPZFUCQN5OMZAHMWDIXGOHYSQ54BQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.728073139 (517h21m44.599156174s) [noise=true , address=145.79.227.52:4447 , piece_id=MCDADETIJYZVDDELUE3OV7K7FYT6M7WTE6PEEJV2PKJSA2UYKCRA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.729538148 (517h21m44.600621193s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.730874213 (517h21m44.601957258s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.733746442 (517h21m44.604829477s) [noise=true , address=145.79.227.26:4654 , piece_id=5L4VCMGH6ZPNMMAACNCLDHJ5ACGTWD7BCFC6R4GIJXALUQOGT5EA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.736813432 (517h21m44.607896467s) [noise=true , address=145.79.227.36:4518 , piece_id=DQ6VYB4X5J6EXPXAGSRAYXMOFGHZJBFJJIINP5WMJNBOLP4HWLCQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:46.349325320 (517h21m46.220408355s) [upload=453922]: Next returned nil segment Dec 27 04:50:46.349563915 (517h21m46.22064695s) [upload=453922]: Waiting for error group managing segments... Dec 27 04:50:46.728761261 (517h21m46.599844306s) [noise=true , address=145.79.227.36:4518 , piece_id=DQ6VYB4X5J6EXPXAGSRAYXMOFGHZJBFJJIINP5WMJNBOLP4HWLCQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.735006383 (517h21m46.606089418s) [noise=true , address=145.79.227.35:4483 , piece_id=XN52OMDOQXQNUNZGZE4RLEN3NPOCJVRPWT2BMF5JNCD6AV4W24YQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.736280198 (517h21m46.607363233s) [noise=true , address=145.79.227.39:4014 , piece_id=5ZIKJPTBAI33C4GCEHMZJWZRA3DWUAYR3YKHEDFFABC5CJDXBJWQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.736925367 (517h21m46.608008412s) [noise=true , address=145.79.227.38:4978 , piece_id=3ALOI3LKLPJINDAPR4UL6STPPZFUCQN5OMZAHMWDIXGOHYSQ54BQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.744323314 (517h21m46.615406349s) [noise=true , address=145.79.227.26:4654 , piece_id=5L4VCMGH6ZPNMMAACNCLDHJ5ACGTWD7BCFC6R4GIJXALUQOGT5EA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.748123350 (517h21m46.619206416s) [noise=true , address=145.79.227.17:4972 , piece_id=3HQMNCQJFBXFB3WA2TKY7F4R7EVF7O3WYSE6RZCRHEAG4VOJ23XQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.759159916 (517h21m46.630242961s) [noise=true , address=145.79.227.2:4114 , piece_id=T7RSPUEOXEABLHS357OMDDY5S47L7Y2VBURFLNM4BEK42F4SKUEQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.762073694 (517h21m46.633156739s) [noise=true , address=145.79.227.4:4185 , piece_id=EHX3JCSZHGKQ3RS5DAH7LINNJO6ERSVN7J7JUHXFSW2HFO546ZXA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.774638660 (517h21m46.645721705s) [noise=true , address=145.79.227.52:4447 , piece_id=MCDADETIJYZVDDELUE3OV7K7FYT6M7WTE6PEEJV2PKJSA2UYKCRA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.989846352 (517h21m46.860929397s) [noise=true , address=45.91.116.6:4974 , piece_id=TVLEEJK6ENP326EF4B6PERDX43NSZ4GFT4OECQ3J55OUCZ7ZY6DQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.052153085 (517h21m46.92323613s) [noise=true , address=77.90.31.21:4880 , piece_id=T22IJQPEDWDCM23NOYUMDMPOQIXGMTVJEOGMSILDZKVK72VWNI3A, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.125333994 (517h21m46.996417029s) [noise=true , address=77.90.31.11:4554 , piece_id=2XBKJBOIIVL5YSTWPJBGKDWOM6K5ZVOAJT7AQZVKJGFPFQEKEQMQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.129239290 (517h21m47.000322335s) [noise=true , address=45.91.116.12:4228 , piece_id=I6QUV2ODXGGCREOKKBKF6XHKSCQIO64ETYUJ4ZBNEQSMQRQB6QGA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.129256633 (517h21m47.000339668s) [seg_pos={0 0}, upload=453922]: Segment reached optimal threshold of 13 pieces. Dec 27 04:50:47.129433039 (517h21m47.000516104s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129442657 (517h21m47.000525702s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129819515 (517h21m47.00090255s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129826208 (517h21m47.000909253s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129954863 (517h21m47.001037898s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129962427 (517h21m47.001045462s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129983497 (517h21m47.001066532s) [seg_pos={0 0}, upload=453922]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:50:47.130050715 (517h21m47.00113375s) [upload=453922]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:50:47.130056877 (517h21m47.001139922s) [upload=453922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:50:47.421324630 (517h21m47.292407675s) [upload=453922]: Done uploading segments... Dec 27 04:50:47.421344989 (517h21m47.292428024s) [upload=453922]: Upload finished. err: <nil> Duration: 13.465672939s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.454048549s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:51:24 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.454137479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [....X.XX..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX] duration: 30.003227982s trace: https://jaeger.dev.storj.io/trace/5e90b03efa41de35 FAIL (2025-12-27 04:51:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 04:50:44.473653280 (517h21m44.344736325s) [upload=453922]: Starting upload Dec 27 04:50:44.473725197 (517h21m44.344808242s) [upload=453922]: Done starting upload Dec 27 04:50:44.473773379 (517h21m44.344856424s) [upload=453922]: Uploading segments... Dec 27 04:50:44.473930388 (517h21m44.345013423s) [upload=453922]: Got next segment. Inline: false Dec 27 04:50:44.473966126 (517h21m44.345049161s) [upload=453922]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:50:44.473970133 (517h21m44.345053168s) [upload=453922]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:50:44.720782557 (517h21m44.591865592s) [seg_pos={0 0}, upload=453922]: Begin upload segment... Dec 27 04:50:44.720855896 (517h21m44.591938931s) [seg_pos={0 0}, upload=453922]: Done begin upload segment. Dec 27 04:50:44.720939585 (517h21m44.59202263s) [noise=true , address=77.90.31.11:4554 , piece_id=2XBKJBOIIVL5YSTWPJBGKDWOM6K5ZVOAJT7AQZVKJGFPFQEKEQMQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.720989631 (517h21m44.592072666s) [noise=true , address=145.79.227.4:4185 , piece_id=EHX3JCSZHGKQ3RS5DAH7LINNJO6ERSVN7J7JUHXFSW2HFO546ZXA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.721643657 (517h21m44.592726692s) [noise=true , address=45.91.116.6:4974 , piece_id=TVLEEJK6ENP326EF4B6PERDX43NSZ4GFT4OECQ3J55OUCZ7ZY6DQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.722233099 (517h21m44.593316134s) [noise=true , address=145.79.227.39:4014 , piece_id=5ZIKJPTBAI33C4GCEHMZJWZRA3DWUAYR3YKHEDFFABC5CJDXBJWQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.722786612 (517h21m44.593869647s) [noise=true , address=145.79.227.17:4972 , piece_id=3HQMNCQJFBXFB3WA2TKY7F4R7EVF7O3WYSE6RZCRHEAG4VOJ23XQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.723175383 (517h21m44.594258418s) [noise=true , address=77.90.31.21:4880 , piece_id=T22IJQPEDWDCM23NOYUMDMPOQIXGMTVJEOGMSILDZKVK72VWNI3A, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.723600533 (517h21m44.594683588s) [noise=true , address=45.91.116.12:4228 , piece_id=I6QUV2ODXGGCREOKKBKF6XHKSCQIO64ETYUJ4ZBNEQSMQRQB6QGA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.724048866 (517h21m44.595131901s) [noise=true , address=145.79.227.35:4483 , piece_id=XN52OMDOQXQNUNZGZE4RLEN3NPOCJVRPWT2BMF5JNCD6AV4W24YQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.725054301 (517h21m44.596137346s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.725518304 (517h21m44.596601349s) [noise=true , address=145.79.227.2:4114 , piece_id=T7RSPUEOXEABLHS357OMDDY5S47L7Y2VBURFLNM4BEK42F4SKUEQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.726666771 (517h21m44.597749816s) [noise=true , address=145.79.227.38:4978 , piece_id=3ALOI3LKLPJINDAPR4UL6STPPZFUCQN5OMZAHMWDIXGOHYSQ54BQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.728073139 (517h21m44.599156174s) [noise=true , address=145.79.227.52:4447 , piece_id=MCDADETIJYZVDDELUE3OV7K7FYT6M7WTE6PEEJV2PKJSA2UYKCRA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.729538148 (517h21m44.600621193s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.730874213 (517h21m44.601957258s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.733746442 (517h21m44.604829477s) [noise=true , address=145.79.227.26:4654 , piece_id=5L4VCMGH6ZPNMMAACNCLDHJ5ACGTWD7BCFC6R4GIJXALUQOGT5EA, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:44.736813432 (517h21m44.607896467s) [noise=true , address=145.79.227.36:4518 , piece_id=DQ6VYB4X5J6EXPXAGSRAYXMOFGHZJBFJJIINP5WMJNBOLP4HWLCQ, seg_pos={0 0}, upload=453922]: Uploading piece... Dec 27 04:50:46.349325320 (517h21m46.220408355s) [upload=453922]: Next returned nil segment Dec 27 04:50:46.349563915 (517h21m46.22064695s) [upload=453922]: Waiting for error group managing segments... Dec 27 04:50:46.728761261 (517h21m46.599844306s) [noise=true , address=145.79.227.36:4518 , piece_id=DQ6VYB4X5J6EXPXAGSRAYXMOFGHZJBFJJIINP5WMJNBOLP4HWLCQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.735006383 (517h21m46.606089418s) [noise=true , address=145.79.227.35:4483 , piece_id=XN52OMDOQXQNUNZGZE4RLEN3NPOCJVRPWT2BMF5JNCD6AV4W24YQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.736280198 (517h21m46.607363233s) [noise=true , address=145.79.227.39:4014 , piece_id=5ZIKJPTBAI33C4GCEHMZJWZRA3DWUAYR3YKHEDFFABC5CJDXBJWQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.736925367 (517h21m46.608008412s) [noise=true , address=145.79.227.38:4978 , piece_id=3ALOI3LKLPJINDAPR4UL6STPPZFUCQN5OMZAHMWDIXGOHYSQ54BQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.744323314 (517h21m46.615406349s) [noise=true , address=145.79.227.26:4654 , piece_id=5L4VCMGH6ZPNMMAACNCLDHJ5ACGTWD7BCFC6R4GIJXALUQOGT5EA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.748123350 (517h21m46.619206416s) [noise=true , address=145.79.227.17:4972 , piece_id=3HQMNCQJFBXFB3WA2TKY7F4R7EVF7O3WYSE6RZCRHEAG4VOJ23XQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.759159916 (517h21m46.630242961s) [noise=true , address=145.79.227.2:4114 , piece_id=T7RSPUEOXEABLHS357OMDDY5S47L7Y2VBURFLNM4BEK42F4SKUEQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.762073694 (517h21m46.633156739s) [noise=true , address=145.79.227.4:4185 , piece_id=EHX3JCSZHGKQ3RS5DAH7LINNJO6ERSVN7J7JUHXFSW2HFO546ZXA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.774638660 (517h21m46.645721705s) [noise=true , address=145.79.227.52:4447 , piece_id=MCDADETIJYZVDDELUE3OV7K7FYT6M7WTE6PEEJV2PKJSA2UYKCRA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:46.989846352 (517h21m46.860929397s) [noise=true , address=45.91.116.6:4974 , piece_id=TVLEEJK6ENP326EF4B6PERDX43NSZ4GFT4OECQ3J55OUCZ7ZY6DQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.052153085 (517h21m46.92323613s) [noise=true , address=77.90.31.21:4880 , piece_id=T22IJQPEDWDCM23NOYUMDMPOQIXGMTVJEOGMSILDZKVK72VWNI3A, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.125333994 (517h21m46.996417029s) [noise=true , address=77.90.31.11:4554 , piece_id=2XBKJBOIIVL5YSTWPJBGKDWOM6K5ZVOAJT7AQZVKJGFPFQEKEQMQ, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.129239290 (517h21m47.000322335s) [noise=true , address=45.91.116.12:4228 , piece_id=I6QUV2ODXGGCREOKKBKF6XHKSCQIO64ETYUJ4ZBNEQSMQRQB6QGA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: <nil> Dec 27 04:50:47.129256633 (517h21m47.000339668s) [seg_pos={0 0}, upload=453922]: Segment reached optimal threshold of 13 pieces. Dec 27 04:50:47.129433039 (517h21m47.000516104s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129442657 (517h21m47.000525702s) [noise=true , address=145.79.227.40:4050 , piece_id=UEYFN7QIW6BJAGBNJ6LOIWHL36G5CPVYUL2AEHTAVUAOD7MV277Q, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129819515 (517h21m47.00090255s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129826208 (517h21m47.000909253s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTR4PTGB2Y7TTR3IDPLIH7LCPLMMZZKFACVWZ357V3XI4MC7WPLA, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129954863 (517h21m47.001037898s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 04:50:47.129962427 (517h21m47.001045462s) [noise=true , address=45.91.116.13:4156 , piece_id=GDLETTOZRBL72BQJ2EUSCJAD4Q3R7NAAQNEVQS2SWENVAIBIASDA, seg_pos={0 0}, upload=453922]: Long tail cancellation due to optimal threshold Dec 27 04:50:47.129983497 (517h21m47.001066532s) [seg_pos={0 0}, upload=453922]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:50:47.130050715 (517h21m47.00113375s) [upload=453922]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:50:47.130056877 (517h21m47.001139922s) [upload=453922]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:50:47.421324630 (517h21m47.292407675s) [upload=453922]: Done uploading segments... Dec 27 04:50:47.421344989 (517h21m47.292428024s) [upload=453922]: Upload finished. err: <nil> Duration: 30.00325352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X......................X..] duration: 52.293010692s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:50:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 52.293044886s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.472203499s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:50:49 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.472239107s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.532276769s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:50:19 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.532305463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.....X.XX..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.X] duration: 30.002664829s trace: https://jaeger.dev.storj.io/trace/64792a53093b7e76 FAIL (2025-12-27 04:49:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: D4TPJBZJNUPIQWNTJYHKWQ6MAHX7EM7LP5BQEO2C7NIPZDYHAHZQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: JIYCLCKCE63IX5CPKMJXBRQJIZITVAO67MTOQMBJDQ5N6WFNJ6HQ) context deadline exceeded ======LOGS====== Dec 27 04:49:29.704245809 (517h20m29.575328845s) [upload=453906]: Starting upload Dec 27 04:49:29.704309079 (517h20m29.575392124s) [upload=453906]: Done starting upload Dec 27 04:49:29.704395244 (517h20m29.575478279s) [upload=453906]: Uploading segments... Dec 27 04:49:29.704489113 (517h20m29.575572158s) [upload=453906]: Got next segment. Inline: false Dec 27 04:49:29.704517075 (517h20m29.57560012s) [upload=453906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:49:29.704521695 (517h20m29.57560473s) [upload=453906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:49:29.951936956 (517h20m29.823020001s) [seg_pos={0 0}, upload=453906]: Begin upload segment... Dec 27 04:49:29.951999415 (517h20m29.82308246s) [seg_pos={0 0}, upload=453906]: Done begin upload segment. Dec 27 04:49:29.952031596 (517h20m29.823114641s) [noise=true , address=145.79.227.46:4602 , piece_id=AZBXEQH2S6CADR66QEBPSZZ3DY7BAUNWZMMNAZPMG3X6B24TX2JQ, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.952045803 (517h20m29.823128848s) [noise=true , address=145.79.227.47:4265 , piece_id=53H2ABPCG46LBGGHCKD6PBCV6IR7JVWQO7FXEVYSUUOG7H7BEWLQ, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.952116718 (517h20m29.823199763s) [noise=true , address=145.79.227.45:4230 , piece_id=ODT66OMGHIXU4A5276PCC4636LBH4WHCHVLJZSU3KJGEN5XA2HFA, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.952743191 (517h20m29.823826226s) [noise=true , address=45.91.116.10:4083 , piece_id=KEXIL7WNWIE6VMSPSZUE47YUISUW4C7RV2FVCH46HHOP4YHEMPEA, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.953154584 (517h20m29.824237629s) [noise=true , address=145.79.227.51:4410 , piece_id=NDOHDYDQ6XD7EZZFW4XQDORDYVVNZHIL44SV5V2FBRQLN5VDG5PQ, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.955219626 (517h20m29.826302671s) [noise=true , address=145.79.227.49:4339 , piece_id=KOC62KYMJHGHCFCF77CVUNR4GX77YGWZJN5O5QUPTZJBC7NMSWVQ, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.956444158 (517h20m29.827527203s) [noise=true , address=145.79.227.15:4875 , piece_id=2Q5PIJUIKHHHOCGPFXAC6BOGR4WJNDUKVGED6PKQWU66GQIOA6LQ, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.956665670 (517h20m29.827748715s) [noise=true , address=145.79.227.13:4802 , piece_id=YM67UTUANWJEDKLC7ZGFUZZE3RJEVJNOEIZFEOXBD67RE5EPMIAA, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.957966166 (517h20m29.829049201s) [noise=true , address=77.90.31.21:4878 , piece_id=6TY25B6GOYV67GFP54J3UJU3FXZIUHTZ6HEQAPGC6NXYOVNKAXOA, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.960048231 (517h20m29.831131266s) [noise=true , address=145.79.227.50:4375 , piece_id=XPIQXOT2JWKH4AFSLTGDVJQ2DCUROGXAWUQTJA2LBJOMTKFMSX4A, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.961833671 (517h20m29.832916716s) [noise=true , address=45.91.116.13:4156 , piece_id=GOTJMEZRLHPJTFZ2LCM2RTUIYVTTJTLURPB4KUGYIAIDGX2H4UDQ, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.962289408 (517h20m29.833372453s) [noise=true , address=77.90.31.13:4629 , piece_id=D4TPJBZJNUPIQWNTJYHKWQ6MAHX7EM7LP5BQEO2C7NIPZDYHAHZQ, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.963985268 (517h20m29.835068313s) [noise=true , address=77.90.31.12:4589 , piece_id=JIYCLCKCE63IX5CPKMJXBRQJIZITVAO67MTOQMBJDQ5N6WFNJ6HQ, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.966680520 (517h20m29.837763566s) [noise=true , address=145.79.227.38:4979 , piece_id=3JIERZYROEROEFQH4L24YGCZ34UYWP43NEIKU7FXFJPL6MA6GO2Q, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.968559447 (517h20m29.839642492s) [noise=true , address=45.91.116.4:4939 , piece_id=N6WD7JB4YPLJR2QBEWVGFTNMZJK6VKEM5TPTUFAVX2KTD327S73Q, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:29.970548937 (517h20m29.841631982s) [noise=true , address=145.79.227.10:4403 , piece_id=VQJRITYNMDOXHWSTSIYEC54MNFVEJPXOGPEMW27Q25MFND3UZRMQ, seg_pos={0 0}, upload=453906]: Uploading piece... Dec 27 04:49:31.668503411 (517h20m31.539586466s) [upload=453906]: Next returned nil segment Dec 27 04:49:31.668531725 (517h20m31.53961478s) [upload=453906]: Waiting for error group managing segments... Dec 27 04:49:31.953411446 (517h20m31.824494491s) [noise=true , address=145.79.227.46:4602 , piece_id=AZBXEQH2S6CADR66QEBPSZZ3DY7BAUNWZMMNAZPMG3X6B24TX2JQ, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:31.961334642 (517h20m31.832417687s) [noise=true , address=145.79.227.51:4410 , piece_id=NDOHDYDQ6XD7EZZFW4XQDORDYVVNZHIL44SV5V2FBRQLN5VDG5PQ, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:31.965188501 (517h20m31.836271546s) [noise=true , address=145.79.227.13:4802 , piece_id=YM67UTUANWJEDKLC7ZGFUZZE3RJEVJNOEIZFEOXBD67RE5EPMIAA, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:31.967346791 (517h20m31.838429836s) [noise=true , address=145.79.227.15:4875 , piece_id=2Q5PIJUIKHHHOCGPFXAC6BOGR4WJNDUKVGED6PKQWU66GQIOA6LQ, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:31.972242684 (517h20m31.843325729s) [noise=true , address=145.79.227.38:4979 , piece_id=3JIERZYROEROEFQH4L24YGCZ34UYWP43NEIKU7FXFJPL6MA6GO2Q, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:31.972575066 (517h20m31.843658101s) [noise=true , address=145.79.227.47:4265 , piece_id=53H2ABPCG46LBGGHCKD6PBCV6IR7JVWQO7FXEVYSUUOG7H7BEWLQ, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:31.977786189 (517h20m31.848869224s) [noise=true , address=145.79.227.49:4339 , piece_id=KOC62KYMJHGHCFCF77CVUNR4GX77YGWZJN5O5QUPTZJBC7NMSWVQ, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:31.982100925 (517h20m31.85318396s) [noise=true , address=145.79.227.45:4230 , piece_id=ODT66OMGHIXU4A5276PCC4636LBH4WHCHVLJZSU3KJGEN5XA2HFA, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:31.983519927 (517h20m31.854602962s) [noise=true , address=145.79.227.50:4375 , piece_id=XPIQXOT2JWKH4AFSLTGDVJQ2DCUROGXAWUQTJA2LBJOMTKFMSX4A, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:31.991462151 (517h20m31.862545196s) [noise=true , address=145.79.227.10:4403 , piece_id=VQJRITYNMDOXHWSTSIYEC54MNFVEJPXOGPEMW27Q25MFND3UZRMQ, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:32.138032785 (517h20m32.00911583s) [noise=true , address=77.90.31.13:4629 , piece_id=D4TPJBZJNUPIQWNTJYHKWQ6MAHX7EM7LP5BQEO2C7NIPZDYHAHZQ, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:32.211646088 (517h20m32.082729123s) [noise=true , address=77.90.31.12:4589 , piece_id=JIYCLCKCE63IX5CPKMJXBRQJIZITVAO67MTOQMBJDQ5N6WFNJ6HQ, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:32.241455110 (517h20m32.112538145s) [noise=true , address=77.90.31.21:4878 , piece_id=6TY25B6GOYV67GFP54J3UJU3FXZIUHTZ6HEQAPGC6NXYOVNKAXOA, seg_pos={0 0}, upload=453906]: Done uploading piece. err: <nil> Dec 27 04:49:32.241473485 (517h20m32.11255652s) [seg_pos={0 0}, upload=453906]: Segment reached optimal threshold of 13 pieces. Dec 27 04:49:32.241653417 (517h20m32.112736452s) [noise=true , address=45.91.116.4:4939 , piece_id=N6WD7JB4YPLJR2QBEWVGFTNMZJK6VKEM5TPTUFAVX2KTD327S73Q, seg_pos={0 0}, upload=453906]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 04:49:32.241661904 (517h20m32.112744949s) [noise=true , address=45.91.116.4:4939 , piece_id=N6WD7JB4YPLJR2QBEWVGFTNMZJK6VKEM5TPTUFAVX2KTD327S73Q, seg_pos={0 0}, upload=453906]: Long tail cancellation due to optimal threshold Dec 27 04:49:32.241916709 (517h20m32.112999754s) [noise=true , address=45.91.116.10:4083 , piece_id=KEXIL7WNWIE6VMSPSZUE47YUISUW4C7RV2FVCH46HHOP4YHEMPEA, seg_pos={0 0}, upload=453906]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 04:49:32.241926627 (517h20m32.113009673s) [noise=true , address=45.91.116.10:4083 , piece_id=KEXIL7WNWIE6VMSPSZUE47YUISUW4C7RV2FVCH46HHOP4YHEMPEA, seg_pos={0 0}, upload=453906]: Long tail cancellation due to optimal threshold Dec 27 04:49:32.241962476 (517h20m32.113045521s) [noise=true , address=45.91.116.13:4156 , piece_id=GOTJMEZRLHPJTFZ2LCM2RTUIYVTTJTLURPB4KUGYIAIDGX2H4UDQ, seg_pos={0 0}, upload=453906]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 04:49:32.241973507 (517h20m32.113056552s) [noise=true , address=45.91.116.13:4156 , piece_id=GOTJMEZRLHPJTFZ2LCM2RTUIYVTTJTLURPB4KUGYIAIDGX2H4UDQ, seg_pos={0 0}, upload=453906]: Long tail cancellation due to optimal threshold Dec 27 04:49:32.242032289 (517h20m32.113115324s) [seg_pos={0 0}, upload=453906]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:49:32.242194177 (517h20m32.113277232s) [upload=453906]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:49:32.242212041 (517h20m32.113295076s) [upload=453906]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:49:32.527901573 (517h20m32.398984618s) [upload=453906]: Done uploading segments... Dec 27 04:49:32.527919657 (517h20m32.399002692s) [upload=453906]: Upload finished. err: <nil> Duration: 30.002690107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X......................X.] duration: 35.454238507s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:49:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 35.45427187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [......X.XX..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.] duration: 13.521376912s trace: https://jaeger.dev.storj.io/trace/307c8391af4390cc FAIL (2025-12-27 04:49:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: YMLDJEXILVYQFVVIA6G5F3H6YF7U4I5XJARFXXHX7RRRWM6JVFZQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: T3HCPCRTMA2LYFW6UT3T3DPMLDDYFJ7MTOVWDGIGNQ7FEF334W2Q) context deadline exceeded ======LOGS====== Dec 27 04:48:01.426243461 (517h19m1.297326506s) [upload=453884]: Starting upload Dec 27 04:48:01.426301591 (517h19m1.297384637s) [upload=453884]: Done starting upload Dec 27 04:48:01.426667378 (517h19m1.297750434s) [upload=453884]: Uploading segments... Dec 27 04:48:01.427013017 (517h19m1.298096052s) [upload=453884]: Got next segment. Inline: false Dec 27 04:48:01.427078201 (517h19m1.298161236s) [upload=453884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:48:01.427085895 (517h19m1.29816895s) [upload=453884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:48:01.671952834 (517h19m1.543035889s) [seg_pos={0 0}, upload=453884]: Begin upload segment... Dec 27 04:48:01.672012417 (517h19m1.543095462s) [seg_pos={0 0}, upload=453884]: Done begin upload segment. Dec 27 04:48:01.672045450 (517h19m1.543128505s) [noise=true , address=45.91.116.13:4155 , piece_id=MUTDYQUHNH2U6AUDNCSJUJJVNV62XRXNXU7AEY2YDNBARBZUKB7A, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.673376124 (517h19m1.544459159s) [noise=true , address=145.79.227.7:4293 , piece_id=3COE7PSCDH3GF3DBJUGKZYRLXANWMC3ZHNRHYONYE7S2TR4OXEPA, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.673943655 (517h19m1.5450267s) [noise=true , address=145.79.227.15:4875 , piece_id=5HCIJZR7NU4Q6GEDHRZFEJJRIVTUJATB47BJO2EQWSH53PMCQPPQ, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.674739641 (517h19m1.545822686s) [noise=true , address=77.90.31.14:4664 , piece_id=KQ67YIX3ID3LUVQRBSBMSKM6TP6JOFFIRK5KVBA6RNOZGR45XJAQ, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.675306099 (517h19m1.546389134s) [noise=true , address=45.91.116.14:4190 , piece_id=LN3NSZ2WLSJ5W6LGEQB5R654RTPFM6B53KCZFHJPYQS3NU3HX7VQ, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.675635196 (517h19m1.546718241s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEC3WBT7EKN5M5ZPSWCQU7BVJRGHMT3LEU4YSBV3AFXZVVO3EA, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.675953442 (517h19m1.547036477s) [noise=true , address=145.79.227.6:4259 , piece_id=SMIWQZJNBCN7CYSM6S5JBHD5NNC7HXJZNUBIMFAQZLRSMO3KIUGA, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.676256468 (517h19m1.547339503s) [noise=true , address=45.91.116.7:4012 , piece_id=KEEONUM4QRJJVS4Z32TYXXPWO6K7CQOLELUADHUQI3F234NJNH7A, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.676637694 (517h19m1.547720729s) [noise=true , address=45.91.116.8:4047 , piece_id=YMLDJEXILVYQFVVIA6G5F3H6YF7U4I5XJARFXXHX7RRRWM6JVFZQ, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.677004734 (517h19m1.548087769s) [noise=true , address=145.79.227.45:4229 , piece_id=J55TL6UX4DKIZYRV7GFKCOGXPNNPU3WTAPF2G2LPYQM2E7FDVYVA, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.677392842 (517h19m1.548475888s) [noise=true , address=145.79.227.13:4802 , piece_id=GII4TXRJFCOG7WRA5Y6RTVP3Y2HSC6G7DWRLHMVDE3DXUVCBLALQ, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.677789016 (517h19m1.548872052s) [noise=true , address=145.79.227.39:4014 , piece_id=VZFDZ4NTZEE6TUKNGOUJ72UNX37OCAMEN3DPYAV4VU24B522NZ7A, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.678273810 (517h19m1.549356855s) [noise=true , address=145.79.227.11:4732 , piece_id=5G42IIG7ZSGCPGCXDZFLQ2KRFZDXOWGFRWEKQIV6HOVFC3JWVWJQ, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.678852873 (517h19m1.549935908s) [noise=true , address=77.90.31.17:4770 , piece_id=CNGGHHJELLY2F2BLSZFJ6UACSSD4TOUVDZNLABCAXCCMQIFXKIYA, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.680292353 (517h19m1.551375398s) [noise=true , address=45.91.116.12:4227 , piece_id=T3HCPCRTMA2LYFW6UT3T3DPMLDDYFJ7MTOVWDGIGNQ7FEF334W2Q, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.681537014 (517h19m1.552620049s) [noise=true , address=145.79.227.31:4799 , piece_id=ETHIOEJDOQGBHBF7HWI2FE3YLKMMEQU7B3ZDDIXRIBECRES7T5GA, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:03.447046716 (517h19m3.318129751s) [upload=453884]: Next returned nil segment Dec 27 04:48:03.447499888 (517h19m3.318582954s) [upload=453884]: Waiting for error group managing segments... Dec 27 04:48:03.856767465 (517h19m3.72785051s) [noise=true , address=77.90.31.17:4770 , piece_id=CNGGHHJELLY2F2BLSZFJ6UACSSD4TOUVDZNLABCAXCCMQIFXKIYA, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:03.951199669 (517h19m3.822282714s) [noise=true , address=45.91.116.7:4012 , piece_id=KEEONUM4QRJJVS4Z32TYXXPWO6K7CQOLELUADHUQI3F234NJNH7A, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:04.000283293 (517h19m3.871366348s) [noise=true , address=45.91.116.8:4047 , piece_id=YMLDJEXILVYQFVVIA6G5F3H6YF7U4I5XJARFXXHX7RRRWM6JVFZQ, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:04.420466524 (517h19m4.291549569s) [noise=true , address=45.91.116.14:4190 , piece_id=LN3NSZ2WLSJ5W6LGEQB5R654RTPFM6B53KCZFHJPYQS3NU3HX7VQ, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:04.602571473 (517h19m4.473654508s) [noise=true , address=77.90.31.14:4664 , piece_id=KQ67YIX3ID3LUVQRBSBMSKM6TP6JOFFIRK5KVBA6RNOZGR45XJAQ, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:04.686673717 (517h19m4.557756762s) [noise=true , address=45.91.116.13:4155 , piece_id=MUTDYQUHNH2U6AUDNCSJUJJVNV62XRXNXU7AEY2YDNBARBZUKB7A, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:05.095264013 (517h19m4.966347058s) [noise=true , address=45.91.116.12:4227 , piece_id=T3HCPCRTMA2LYFW6UT3T3DPMLDDYFJ7MTOVWDGIGNQ7FEF334W2Q, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:06.000062213 (517h19m5.871145258s) [noise=true , address=145.79.227.7:4293 , piece_id=3COE7PSCDH3GF3DBJUGKZYRLXANWMC3ZHNRHYONYE7S2TR4OXEPA, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:07.194938632 (517h19m7.066021677s) [noise=true , address=145.79.227.45:4229 , piece_id=J55TL6UX4DKIZYRV7GFKCOGXPNNPU3WTAPF2G2LPYQM2E7FDVYVA, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:07.514185543 (517h19m7.385268578s) [noise=true , address=145.79.227.11:4732 , piece_id=5G42IIG7ZSGCPGCXDZFLQ2KRFZDXOWGFRWEKQIV6HOVFC3JWVWJQ, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:09.107108160 (517h19m8.978191195s) [noise=true , address=145.79.227.39:4014 , piece_id=VZFDZ4NTZEE6TUKNGOUJ72UNX37OCAMEN3DPYAV4VU24B522NZ7A, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:16.221099144 (517h19m16.092182179s) [noise=true , address=145.79.227.13:4802 , piece_id=GII4TXRJFCOG7WRA5Y6RTVP3Y2HSC6G7DWRLHMVDE3DXUVCBLALQ, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:18.735756251 (517h19m18.606839286s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEC3WBT7EKN5M5ZPSWCQU7BVJRGHMT3LEU4YSBV3AFXZVVO3EA, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:18.735780828 (517h19m18.606863863s) [seg_pos={0 0}, upload=453884]: Segment reached optimal threshold of 13 pieces. Dec 27 04:48:18.736098593 (517h19m18.607181638s) [noise=true , address=145.79.227.31:4799 , piece_id=ETHIOEJDOQGBHBF7HWI2FE3YLKMMEQU7B3ZDDIXRIBECRES7T5GA, seg_pos={0 0}, upload=453884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: read tcp 188.166.225.170:51928->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:18.736106337 (517h19m18.607189382s) [noise=true , address=145.79.227.31:4799 , piece_id=ETHIOEJDOQGBHBF7HWI2FE3YLKMMEQU7B3ZDDIXRIBECRES7T5GA, seg_pos={0 0}, upload=453884]: Long tail cancellation due to optimal threshold Dec 27 04:48:18.736120154 (517h19m18.607203189s) [noise=true , address=145.79.227.6:4259 , piece_id=SMIWQZJNBCN7CYSM6S5JBHD5NNC7HXJZNUBIMFAQZLRSMO3KIUGA, seg_pos={0 0}, upload=453884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: read tcp 188.166.225.170:46102->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:18.736128350 (517h19m18.607211385s) [noise=true , address=145.79.227.6:4259 , piece_id=SMIWQZJNBCN7CYSM6S5JBHD5NNC7HXJZNUBIMFAQZLRSMO3KIUGA, seg_pos={0 0}, upload=453884]: Long tail cancellation due to optimal threshold Dec 27 04:48:18.736181049 (517h19m18.607264094s) [noise=true , address=145.79.227.15:4875 , piece_id=5HCIJZR7NU4Q6GEDHRZFEJJRIVTUJATB47BJO2EQWSH53PMCQPPQ, seg_pos={0 0}, upload=453884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: context canceled; context canceled Dec 27 04:48:18.736186220 (517h19m18.607269265s) [noise=true , address=145.79.227.15:4875 , piece_id=5HCIJZR7NU4Q6GEDHRZFEJJRIVTUJATB47BJO2EQWSH53PMCQPPQ, seg_pos={0 0}, upload=453884]: Long tail cancellation due to optimal threshold Dec 27 04:48:18.736202451 (517h19m18.607285486s) [seg_pos={0 0}, upload=453884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:48:18.736271372 (517h19m18.607354417s) [upload=453884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:18.736276501 (517h19m18.607359536s) [upload=453884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:19.025600464 (517h19m18.896683499s) [upload=453884]: Done uploading segments... Dec 27 04:48:19.025621754 (517h19m18.896704789s) [upload=453884]: Upload finished. err: <nil> Duration: 13.521397351s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.582859327s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:49:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.582889544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...................................X......................X] duration: 43.253040589s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:49:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 43.253081586s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.630574082s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:48:52 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.630600764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.......X.XX..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X] duration: 30.002147175s trace: https://jaeger.dev.storj.io/trace/307c8391af4390cc FAIL (2025-12-27 04:48:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: YMLDJEXILVYQFVVIA6G5F3H6YF7U4I5XJARFXXHX7RRRWM6JVFZQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: T3HCPCRTMA2LYFW6UT3T3DPMLDDYFJ7MTOVWDGIGNQ7FEF334W2Q) context deadline exceeded ======LOGS====== Dec 27 04:48:01.426243461 (517h19m1.297326506s) [upload=453884]: Starting upload Dec 27 04:48:01.426301591 (517h19m1.297384637s) [upload=453884]: Done starting upload Dec 27 04:48:01.426667378 (517h19m1.297750434s) [upload=453884]: Uploading segments... Dec 27 04:48:01.427013017 (517h19m1.298096052s) [upload=453884]: Got next segment. Inline: false Dec 27 04:48:01.427078201 (517h19m1.298161236s) [upload=453884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:48:01.427085895 (517h19m1.29816895s) [upload=453884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:48:01.671952834 (517h19m1.543035889s) [seg_pos={0 0}, upload=453884]: Begin upload segment... Dec 27 04:48:01.672012417 (517h19m1.543095462s) [seg_pos={0 0}, upload=453884]: Done begin upload segment. Dec 27 04:48:01.672045450 (517h19m1.543128505s) [noise=true , address=45.91.116.13:4155 , piece_id=MUTDYQUHNH2U6AUDNCSJUJJVNV62XRXNXU7AEY2YDNBARBZUKB7A, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.673376124 (517h19m1.544459159s) [noise=true , address=145.79.227.7:4293 , piece_id=3COE7PSCDH3GF3DBJUGKZYRLXANWMC3ZHNRHYONYE7S2TR4OXEPA, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.673943655 (517h19m1.5450267s) [noise=true , address=145.79.227.15:4875 , piece_id=5HCIJZR7NU4Q6GEDHRZFEJJRIVTUJATB47BJO2EQWSH53PMCQPPQ, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.674739641 (517h19m1.545822686s) [noise=true , address=77.90.31.14:4664 , piece_id=KQ67YIX3ID3LUVQRBSBMSKM6TP6JOFFIRK5KVBA6RNOZGR45XJAQ, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.675306099 (517h19m1.546389134s) [noise=true , address=45.91.116.14:4190 , piece_id=LN3NSZ2WLSJ5W6LGEQB5R654RTPFM6B53KCZFHJPYQS3NU3HX7VQ, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.675635196 (517h19m1.546718241s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEC3WBT7EKN5M5ZPSWCQU7BVJRGHMT3LEU4YSBV3AFXZVVO3EA, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.675953442 (517h19m1.547036477s) [noise=true , address=145.79.227.6:4259 , piece_id=SMIWQZJNBCN7CYSM6S5JBHD5NNC7HXJZNUBIMFAQZLRSMO3KIUGA, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.676256468 (517h19m1.547339503s) [noise=true , address=45.91.116.7:4012 , piece_id=KEEONUM4QRJJVS4Z32TYXXPWO6K7CQOLELUADHUQI3F234NJNH7A, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.676637694 (517h19m1.547720729s) [noise=true , address=45.91.116.8:4047 , piece_id=YMLDJEXILVYQFVVIA6G5F3H6YF7U4I5XJARFXXHX7RRRWM6JVFZQ, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.677004734 (517h19m1.548087769s) [noise=true , address=145.79.227.45:4229 , piece_id=J55TL6UX4DKIZYRV7GFKCOGXPNNPU3WTAPF2G2LPYQM2E7FDVYVA, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.677392842 (517h19m1.548475888s) [noise=true , address=145.79.227.13:4802 , piece_id=GII4TXRJFCOG7WRA5Y6RTVP3Y2HSC6G7DWRLHMVDE3DXUVCBLALQ, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.677789016 (517h19m1.548872052s) [noise=true , address=145.79.227.39:4014 , piece_id=VZFDZ4NTZEE6TUKNGOUJ72UNX37OCAMEN3DPYAV4VU24B522NZ7A, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.678273810 (517h19m1.549356855s) [noise=true , address=145.79.227.11:4732 , piece_id=5G42IIG7ZSGCPGCXDZFLQ2KRFZDXOWGFRWEKQIV6HOVFC3JWVWJQ, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.678852873 (517h19m1.549935908s) [noise=true , address=77.90.31.17:4770 , piece_id=CNGGHHJELLY2F2BLSZFJ6UACSSD4TOUVDZNLABCAXCCMQIFXKIYA, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.680292353 (517h19m1.551375398s) [noise=true , address=45.91.116.12:4227 , piece_id=T3HCPCRTMA2LYFW6UT3T3DPMLDDYFJ7MTOVWDGIGNQ7FEF334W2Q, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:01.681537014 (517h19m1.552620049s) [noise=true , address=145.79.227.31:4799 , piece_id=ETHIOEJDOQGBHBF7HWI2FE3YLKMMEQU7B3ZDDIXRIBECRES7T5GA, seg_pos={0 0}, upload=453884]: Uploading piece... Dec 27 04:48:03.447046716 (517h19m3.318129751s) [upload=453884]: Next returned nil segment Dec 27 04:48:03.447499888 (517h19m3.318582954s) [upload=453884]: Waiting for error group managing segments... Dec 27 04:48:03.856767465 (517h19m3.72785051s) [noise=true , address=77.90.31.17:4770 , piece_id=CNGGHHJELLY2F2BLSZFJ6UACSSD4TOUVDZNLABCAXCCMQIFXKIYA, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:03.951199669 (517h19m3.822282714s) [noise=true , address=45.91.116.7:4012 , piece_id=KEEONUM4QRJJVS4Z32TYXXPWO6K7CQOLELUADHUQI3F234NJNH7A, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:04.000283293 (517h19m3.871366348s) [noise=true , address=45.91.116.8:4047 , piece_id=YMLDJEXILVYQFVVIA6G5F3H6YF7U4I5XJARFXXHX7RRRWM6JVFZQ, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:04.420466524 (517h19m4.291549569s) [noise=true , address=45.91.116.14:4190 , piece_id=LN3NSZ2WLSJ5W6LGEQB5R654RTPFM6B53KCZFHJPYQS3NU3HX7VQ, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:04.602571473 (517h19m4.473654508s) [noise=true , address=77.90.31.14:4664 , piece_id=KQ67YIX3ID3LUVQRBSBMSKM6TP6JOFFIRK5KVBA6RNOZGR45XJAQ, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:04.686673717 (517h19m4.557756762s) [noise=true , address=45.91.116.13:4155 , piece_id=MUTDYQUHNH2U6AUDNCSJUJJVNV62XRXNXU7AEY2YDNBARBZUKB7A, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:05.095264013 (517h19m4.966347058s) [noise=true , address=45.91.116.12:4227 , piece_id=T3HCPCRTMA2LYFW6UT3T3DPMLDDYFJ7MTOVWDGIGNQ7FEF334W2Q, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:06.000062213 (517h19m5.871145258s) [noise=true , address=145.79.227.7:4293 , piece_id=3COE7PSCDH3GF3DBJUGKZYRLXANWMC3ZHNRHYONYE7S2TR4OXEPA, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:07.194938632 (517h19m7.066021677s) [noise=true , address=145.79.227.45:4229 , piece_id=J55TL6UX4DKIZYRV7GFKCOGXPNNPU3WTAPF2G2LPYQM2E7FDVYVA, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:07.514185543 (517h19m7.385268578s) [noise=true , address=145.79.227.11:4732 , piece_id=5G42IIG7ZSGCPGCXDZFLQ2KRFZDXOWGFRWEKQIV6HOVFC3JWVWJQ, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:09.107108160 (517h19m8.978191195s) [noise=true , address=145.79.227.39:4014 , piece_id=VZFDZ4NTZEE6TUKNGOUJ72UNX37OCAMEN3DPYAV4VU24B522NZ7A, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:16.221099144 (517h19m16.092182179s) [noise=true , address=145.79.227.13:4802 , piece_id=GII4TXRJFCOG7WRA5Y6RTVP3Y2HSC6G7DWRLHMVDE3DXUVCBLALQ, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:18.735756251 (517h19m18.606839286s) [noise=true , address=145.79.227.23:4546 , piece_id=5WSEC3WBT7EKN5M5ZPSWCQU7BVJRGHMT3LEU4YSBV3AFXZVVO3EA, seg_pos={0 0}, upload=453884]: Done uploading piece. err: <nil> Dec 27 04:48:18.735780828 (517h19m18.606863863s) [seg_pos={0 0}, upload=453884]: Segment reached optimal threshold of 13 pieces. Dec 27 04:48:18.736098593 (517h19m18.607181638s) [noise=true , address=145.79.227.31:4799 , piece_id=ETHIOEJDOQGBHBF7HWI2FE3YLKMMEQU7B3ZDDIXRIBECRES7T5GA, seg_pos={0 0}, upload=453884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: read tcp 188.166.225.170:51928->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:18.736106337 (517h19m18.607189382s) [noise=true , address=145.79.227.31:4799 , piece_id=ETHIOEJDOQGBHBF7HWI2FE3YLKMMEQU7B3ZDDIXRIBECRES7T5GA, seg_pos={0 0}, upload=453884]: Long tail cancellation due to optimal threshold Dec 27 04:48:18.736120154 (517h19m18.607203189s) [noise=true , address=145.79.227.6:4259 , piece_id=SMIWQZJNBCN7CYSM6S5JBHD5NNC7HXJZNUBIMFAQZLRSMO3KIUGA, seg_pos={0 0}, upload=453884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: read tcp 188.166.225.170:46102->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:18.736128350 (517h19m18.607211385s) [noise=true , address=145.79.227.6:4259 , piece_id=SMIWQZJNBCN7CYSM6S5JBHD5NNC7HXJZNUBIMFAQZLRSMO3KIUGA, seg_pos={0 0}, upload=453884]: Long tail cancellation due to optimal threshold Dec 27 04:48:18.736181049 (517h19m18.607264094s) [noise=true , address=145.79.227.15:4875 , piece_id=5HCIJZR7NU4Q6GEDHRZFEJJRIVTUJATB47BJO2EQWSH53PMCQPPQ, seg_pos={0 0}, upload=453884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: context canceled; context canceled Dec 27 04:48:18.736186220 (517h19m18.607269265s) [noise=true , address=145.79.227.15:4875 , piece_id=5HCIJZR7NU4Q6GEDHRZFEJJRIVTUJATB47BJO2EQWSH53PMCQPPQ, seg_pos={0 0}, upload=453884]: Long tail cancellation due to optimal threshold Dec 27 04:48:18.736202451 (517h19m18.607285486s) [seg_pos={0 0}, upload=453884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:48:18.736271372 (517h19m18.607354417s) [upload=453884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:18.736276501 (517h19m18.607359536s) [upload=453884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:19.025600464 (517h19m18.896683499s) [upload=453884]: Done uploading segments... Dec 27 04:48:19.025621754 (517h19m18.896704789s) [upload=453884]: Upload finished. err: <nil> Duration: 30.00216068s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.469413107s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:48:23 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.469489533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...................................X......................] duration: 45.118275827s trace: https://jaeger.dev.storj.io/trace/c87a834e8ef35c6 FAIL (2025-12-27 04:48:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, Piece ID: XUT6GYZRI6INV47H5JRO45PTZNNP63TJHONJWZ7ZX2C57K5BKGUQ) context deadline exceeded; piecestore: (Node ID: 1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz, Piece ID: 77VJGA5VAWH2SJWFAXIVGMGCWNO2KOCGTA45TKQDY23HEXBJSBQQ) context deadline exceeded; piecestore: (Node ID: 12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, Piece ID: J6ZYWC5MQE5UTGCDELO6OLH3SVEKFJUCEI5DNASQVLCPK6TNFSCQ) context deadline exceeded ======LOGS====== 27 04:27:06.503245535 (516h58m6.37432857s) [noise=true , address=45.91.116.7:4043 , piece_id=KC5UOUJS2Z3C5H3RF3RD4L6CSQEFBKZ6DBO24IKBDO43Z7B4KQQA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.503611160 (516h58m6.374694196s) [noise=true , address=77.90.31.15:4713 , piece_id=XUT6GYZRI6INV47H5JRO45PTZNNP63TJHONJWZ7ZX2C57K5BKGUQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.503968061 (516h58m6.375051106s) [noise=true , address=77.90.31.14:4678 , piece_id=PXSV7HHXMWBVPCJWXHDZTMPEVSGZSOOKRQE4NCNKJTXTDDCLSTAQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.504256129 (516h58m6.375339164s) [noise=true , address=64.49.13.169:4707 , piece_id=MCDCXRNL3UDPSMA7Z2OKFEIGE323WOO6NBLAHYOJ662ZQNJOAEFQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.504572461 (516h58m6.375655506s) [noise=true , address=145.79.227.46:4618 , piece_id=MCGD45OCXA2MSO5ESYDYJBY3WELG5IVFYVAEPKK4O2DDGXLN3BHQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.504944089 (516h58m6.376027124s) [noise=true , address=77.90.31.19:4862 , piece_id=77VJGA5VAWH2SJWFAXIVGMGCWNO2KOCGTA45TKQDY23HEXBJSBQQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.505755795 (516h58m6.37683883s) [noise=true , address=145.79.227.44:4226 , piece_id=BAVZROOB2AZAYV4BXQBMVIOQTYUSQOQIY4IXB6Z2YDXPATLRZD5Q, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.506138724 (516h58m6.377221759s) [noise=true , address=77.90.31.21:4891 , piece_id=XQV3KPGFJCOEYSWP5IYVS43PFNG4OXUS3EQA3MPO4HYBFA54YZMQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.506583922 (516h58m6.377666957s) [noise=true , address=45.91.116.11:4144 , piece_id=J6ZYWC5MQE5UTGCDELO6OLH3SVEKFJUCEI5DNASQVLCPK6TNFSCQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.506989984 (516h58m6.378073029s) [noise=true , address=145.79.227.40:4063 , piece_id=LNBR4YX5H6QCXVVKNGS7DHAPXUTPHG454PBQX5TN6F5S5BL3GHOA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.507382012 (516h58m6.378465057s) [noise=true , address=145.79.227.34:4918 , piece_id=6LSHYKYLMULRSTG6AQIQZESIG6CJXXCWX454N4GSTQSYR3MFJSBA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.507740744 (516h58m6.378823779s) [noise=true , address=145.79.227.30:4780 , piece_id=HMAX5PBRZU6Z7FAGCOYBNNVJ346FNMMCZXAAVUFUXVLAPQR7DQWA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.508083948 (516h58m6.379166993s) [noise=true , address=145.79.227.33:4890 , piece_id=EMRUMNQFNZUI7C5SPOBEXGCTZVOGZHMVL4YMMBBEX5KSN6HWOK2A, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.508422152 (516h58m6.379505187s) [noise=true , address=45.91.116.13:4163 , piece_id=NHNL5MZQPPINCZ5RGQADENPVBER22OUPWYV5T2M7JIA4OSHTGNXA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.508791535 (516h58m6.37987457s) [noise=true , address=77.90.31.12:4594 , piece_id=KOUSDJU2LDCXR33E5GHU4PDPYSP2AJCBLSQMOCEDERVTQLZHQF3Q, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.509145138 (516h58m6.380228183s) [noise=true , address=77.90.31.18:4812 , piece_id=XQTKFLX5AH7FFMBSW6PWMOMBVI4AWYYOXU6L25ANH6LWUDLWMC5Q, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:07.986779004 (516h58m7.85786204s) [upload=453579]: Next returned nil segment Dec 27 04:27:07.986792661 (516h58m7.857875696s) [upload=453579]: Waiting for error group managing segments... Dec 27 04:27:08.446303300 (516h58m8.317386335s) [noise=true , address=145.79.227.46:4618 , piece_id=MCGD45OCXA2MSO5ESYDYJBY3WELG5IVFYVAEPKK4O2DDGXLN3BHQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.543513967 (516h58m8.414597012s) [noise=true , address=77.90.31.19:4862 , piece_id=77VJGA5VAWH2SJWFAXIVGMGCWNO2KOCGTA45TKQDY23HEXBJSBQQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.572304228 (516h58m8.443387263s) [noise=true , address=77.90.31.16:4767 , piece_id=WBRSJ6RPJKEGFX7SN3GM3E2RS3OOFNGAWN4HCOQRSGYKXB62SWOA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.665325697 (516h58m8.536408742s) [noise=true , address=45.91.116.7:4043 , piece_id=KC5UOUJS2Z3C5H3RF3RD4L6CSQEFBKZ6DBO24IKBDO43Z7B4KQQA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.683431093 (516h58m8.554514138s) [noise=true , address=77.90.31.18:4812 , piece_id=XQTKFLX5AH7FFMBSW6PWMOMBVI4AWYYOXU6L25ANH6LWUDLWMC5Q, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.975168534 (516h58m8.846251579s) [noise=true , address=77.90.31.11:4563 , piece_id=LZKRVINZ2CVQ2IBNIPGRMLFRMHRQK7MANSL267MIQK26SDNIZCDQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.988860225 (516h58m8.85994326s) [noise=true , address=45.91.116.11:4144 , piece_id=J6ZYWC5MQE5UTGCDELO6OLH3SVEKFJUCEI5DNASQVLCPK6TNFSCQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.296067601 (516h58m9.167150646s) [noise=true , address=45.91.116.13:4163 , piece_id=NHNL5MZQPPINCZ5RGQADENPVBER22OUPWYV5T2M7JIA4OSHTGNXA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.300138683 (516h58m9.171221719s) [noise=true , address=145.79.227.34:4918 , piece_id=6LSHYKYLMULRSTG6AQIQZESIG6CJXXCWX454N4GSTQSYR3MFJSBA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.308910276 (516h58m9.179993321s) [noise=true , address=145.79.227.30:4780 , piece_id=HMAX5PBRZU6Z7FAGCOYBNNVJ346FNMMCZXAAVUFUXVLAPQR7DQWA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.338300491 (516h58m9.209383526s) [noise=true , address=145.79.227.44:4226 , piece_id=BAVZROOB2AZAYV4BXQBMVIOQTYUSQOQIY4IXB6Z2YDXPATLRZD5Q, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.425622296 (516h58m9.296705331s) [noise=true , address=145.79.227.33:4890 , piece_id=EMRUMNQFNZUI7C5SPOBEXGCTZVOGZHMVL4YMMBBEX5KSN6HWOK2A, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.868436633 (516h58m9.739519709s) [noise=true , address=77.90.31.15:4713 , piece_id=XUT6GYZRI6INV47H5JRO45PTZNNP63TJHONJWZ7ZX2C57K5BKGUQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.956938287 (516h58m9.828021342s) [noise=true , address=145.79.227.40:4063 , piece_id=LNBR4YX5H6QCXVVKNGS7DHAPXUTPHG454PBQX5TN6F5S5BL3GHOA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.956959947 (516h58m9.828042982s) [seg_pos={0 0}, upload=453579]: Segment reached optimal threshold of 14 pieces. Dec 27 04:27:09.957193332 (516h58m9.828276367s) [noise=true , address=77.90.31.14:4678 , piece_id=PXSV7HHXMWBVPCJWXHDZTMPEVSGZSOOKRQE4NCNKJTXTDDCLSTAQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: expected piece hash; context canceled; EOF Dec 27 04:27:09.957200034 (516h58m9.828283069s) [noise=true , address=77.90.31.14:4678 , piece_id=PXSV7HHXMWBVPCJWXHDZTMPEVSGZSOOKRQE4NCNKJTXTDDCLSTAQ, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957361282 (516h58m9.828444327s) [noise=true , address=77.90.31.12:4594 , piece_id=KOUSDJU2LDCXR33E5GHU4PDPYSP2AJCBLSQMOCEDERVTQLZHQF3Q, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Dec 27 04:27:09.957367413 (516h58m9.828450448s) [noise=true , address=77.90.31.12:4594 , piece_id=KOUSDJU2LDCXR33E5GHU4PDPYSP2AJCBLSQMOCEDERVTQLZHQF3Q, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957376080 (516h58m9.828459125s) [noise=true , address=64.49.13.169:4707 , piece_id=MCDCXRNL3UDPSMA7Z2OKFEIGE323WOO6NBLAHYOJ662ZQNJOAEFQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48910->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48910->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48910->64.49.13.169:4707: use of closed network connection Dec 27 04:27:09.957387151 (516h58m9.828470196s) [noise=true , address=64.49.13.169:4707 , piece_id=MCDCXRNL3UDPSMA7Z2OKFEIGE323WOO6NBLAHYOJ662ZQNJOAEFQ, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957643309 (516h58m9.828726354s) [noise=true , address=77.90.31.21:4891 , piece_id=XQV3KPGFJCOEYSWP5IYVS43PFNG4OXUS3EQA3MPO4HYBFA54YZMQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g): protocol: expected piece hash; context canceled; EOF Dec 27 04:27:09.957651324 (516h58m9.828734359s) [noise=true , address=77.90.31.21:4891 , piece_id=XQV3KPGFJCOEYSWP5IYVS43PFNG4OXUS3EQA3MPO4HYBFA54YZMQ, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957693745 (516h58m9.82877679s) [seg_pos={0 0}, upload=453579]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:27:09.957822199 (516h58m9.828905234s) [upload=453579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:27:09.957828862 (516h58m9.828911907s) [upload=453579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:27:10.257090208 (516h58m10.128173253s) [upload=453579]: Done uploading segments... Dec 27 04:27:10.257117090 (516h58m10.128200135s) [upload=453579]: Upload finished. err: <nil> Duration: 45.118300625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.451360561s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:47:51 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.451385449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...................................X.....................] duration: 36.381540209s trace: https://jaeger.dev.storj.io/trace/c87a834e8ef35c6 FAIL (2025-12-27 04:47:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, Piece ID: XUT6GYZRI6INV47H5JRO45PTZNNP63TJHONJWZ7ZX2C57K5BKGUQ) context deadline exceeded; piecestore: (Node ID: 1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz, Piece ID: 77VJGA5VAWH2SJWFAXIVGMGCWNO2KOCGTA45TKQDY23HEXBJSBQQ) context deadline exceeded; piecestore: (Node ID: 12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, Piece ID: J6ZYWC5MQE5UTGCDELO6OLH3SVEKFJUCEI5DNASQVLCPK6TNFSCQ) context deadline exceeded ======LOGS====== 27 04:27:06.503245535 (516h58m6.37432857s) [noise=true , address=45.91.116.7:4043 , piece_id=KC5UOUJS2Z3C5H3RF3RD4L6CSQEFBKZ6DBO24IKBDO43Z7B4KQQA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.503611160 (516h58m6.374694196s) [noise=true , address=77.90.31.15:4713 , piece_id=XUT6GYZRI6INV47H5JRO45PTZNNP63TJHONJWZ7ZX2C57K5BKGUQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.503968061 (516h58m6.375051106s) [noise=true , address=77.90.31.14:4678 , piece_id=PXSV7HHXMWBVPCJWXHDZTMPEVSGZSOOKRQE4NCNKJTXTDDCLSTAQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.504256129 (516h58m6.375339164s) [noise=true , address=64.49.13.169:4707 , piece_id=MCDCXRNL3UDPSMA7Z2OKFEIGE323WOO6NBLAHYOJ662ZQNJOAEFQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.504572461 (516h58m6.375655506s) [noise=true , address=145.79.227.46:4618 , piece_id=MCGD45OCXA2MSO5ESYDYJBY3WELG5IVFYVAEPKK4O2DDGXLN3BHQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.504944089 (516h58m6.376027124s) [noise=true , address=77.90.31.19:4862 , piece_id=77VJGA5VAWH2SJWFAXIVGMGCWNO2KOCGTA45TKQDY23HEXBJSBQQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.505755795 (516h58m6.37683883s) [noise=true , address=145.79.227.44:4226 , piece_id=BAVZROOB2AZAYV4BXQBMVIOQTYUSQOQIY4IXB6Z2YDXPATLRZD5Q, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.506138724 (516h58m6.377221759s) [noise=true , address=77.90.31.21:4891 , piece_id=XQV3KPGFJCOEYSWP5IYVS43PFNG4OXUS3EQA3MPO4HYBFA54YZMQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.506583922 (516h58m6.377666957s) [noise=true , address=45.91.116.11:4144 , piece_id=J6ZYWC5MQE5UTGCDELO6OLH3SVEKFJUCEI5DNASQVLCPK6TNFSCQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.506989984 (516h58m6.378073029s) [noise=true , address=145.79.227.40:4063 , piece_id=LNBR4YX5H6QCXVVKNGS7DHAPXUTPHG454PBQX5TN6F5S5BL3GHOA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.507382012 (516h58m6.378465057s) [noise=true , address=145.79.227.34:4918 , piece_id=6LSHYKYLMULRSTG6AQIQZESIG6CJXXCWX454N4GSTQSYR3MFJSBA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.507740744 (516h58m6.378823779s) [noise=true , address=145.79.227.30:4780 , piece_id=HMAX5PBRZU6Z7FAGCOYBNNVJ346FNMMCZXAAVUFUXVLAPQR7DQWA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.508083948 (516h58m6.379166993s) [noise=true , address=145.79.227.33:4890 , piece_id=EMRUMNQFNZUI7C5SPOBEXGCTZVOGZHMVL4YMMBBEX5KSN6HWOK2A, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.508422152 (516h58m6.379505187s) [noise=true , address=45.91.116.13:4163 , piece_id=NHNL5MZQPPINCZ5RGQADENPVBER22OUPWYV5T2M7JIA4OSHTGNXA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.508791535 (516h58m6.37987457s) [noise=true , address=77.90.31.12:4594 , piece_id=KOUSDJU2LDCXR33E5GHU4PDPYSP2AJCBLSQMOCEDERVTQLZHQF3Q, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.509145138 (516h58m6.380228183s) [noise=true , address=77.90.31.18:4812 , piece_id=XQTKFLX5AH7FFMBSW6PWMOMBVI4AWYYOXU6L25ANH6LWUDLWMC5Q, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:07.986779004 (516h58m7.85786204s) [upload=453579]: Next returned nil segment Dec 27 04:27:07.986792661 (516h58m7.857875696s) [upload=453579]: Waiting for error group managing segments... Dec 27 04:27:08.446303300 (516h58m8.317386335s) [noise=true , address=145.79.227.46:4618 , piece_id=MCGD45OCXA2MSO5ESYDYJBY3WELG5IVFYVAEPKK4O2DDGXLN3BHQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.543513967 (516h58m8.414597012s) [noise=true , address=77.90.31.19:4862 , piece_id=77VJGA5VAWH2SJWFAXIVGMGCWNO2KOCGTA45TKQDY23HEXBJSBQQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.572304228 (516h58m8.443387263s) [noise=true , address=77.90.31.16:4767 , piece_id=WBRSJ6RPJKEGFX7SN3GM3E2RS3OOFNGAWN4HCOQRSGYKXB62SWOA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.665325697 (516h58m8.536408742s) [noise=true , address=45.91.116.7:4043 , piece_id=KC5UOUJS2Z3C5H3RF3RD4L6CSQEFBKZ6DBO24IKBDO43Z7B4KQQA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.683431093 (516h58m8.554514138s) [noise=true , address=77.90.31.18:4812 , piece_id=XQTKFLX5AH7FFMBSW6PWMOMBVI4AWYYOXU6L25ANH6LWUDLWMC5Q, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.975168534 (516h58m8.846251579s) [noise=true , address=77.90.31.11:4563 , piece_id=LZKRVINZ2CVQ2IBNIPGRMLFRMHRQK7MANSL267MIQK26SDNIZCDQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.988860225 (516h58m8.85994326s) [noise=true , address=45.91.116.11:4144 , piece_id=J6ZYWC5MQE5UTGCDELO6OLH3SVEKFJUCEI5DNASQVLCPK6TNFSCQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.296067601 (516h58m9.167150646s) [noise=true , address=45.91.116.13:4163 , piece_id=NHNL5MZQPPINCZ5RGQADENPVBER22OUPWYV5T2M7JIA4OSHTGNXA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.300138683 (516h58m9.171221719s) [noise=true , address=145.79.227.34:4918 , piece_id=6LSHYKYLMULRSTG6AQIQZESIG6CJXXCWX454N4GSTQSYR3MFJSBA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.308910276 (516h58m9.179993321s) [noise=true , address=145.79.227.30:4780 , piece_id=HMAX5PBRZU6Z7FAGCOYBNNVJ346FNMMCZXAAVUFUXVLAPQR7DQWA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.338300491 (516h58m9.209383526s) [noise=true , address=145.79.227.44:4226 , piece_id=BAVZROOB2AZAYV4BXQBMVIOQTYUSQOQIY4IXB6Z2YDXPATLRZD5Q, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.425622296 (516h58m9.296705331s) [noise=true , address=145.79.227.33:4890 , piece_id=EMRUMNQFNZUI7C5SPOBEXGCTZVOGZHMVL4YMMBBEX5KSN6HWOK2A, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.868436633 (516h58m9.739519709s) [noise=true , address=77.90.31.15:4713 , piece_id=XUT6GYZRI6INV47H5JRO45PTZNNP63TJHONJWZ7ZX2C57K5BKGUQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.956938287 (516h58m9.828021342s) [noise=true , address=145.79.227.40:4063 , piece_id=LNBR4YX5H6QCXVVKNGS7DHAPXUTPHG454PBQX5TN6F5S5BL3GHOA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.956959947 (516h58m9.828042982s) [seg_pos={0 0}, upload=453579]: Segment reached optimal threshold of 14 pieces. Dec 27 04:27:09.957193332 (516h58m9.828276367s) [noise=true , address=77.90.31.14:4678 , piece_id=PXSV7HHXMWBVPCJWXHDZTMPEVSGZSOOKRQE4NCNKJTXTDDCLSTAQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: expected piece hash; context canceled; EOF Dec 27 04:27:09.957200034 (516h58m9.828283069s) [noise=true , address=77.90.31.14:4678 , piece_id=PXSV7HHXMWBVPCJWXHDZTMPEVSGZSOOKRQE4NCNKJTXTDDCLSTAQ, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957361282 (516h58m9.828444327s) [noise=true , address=77.90.31.12:4594 , piece_id=KOUSDJU2LDCXR33E5GHU4PDPYSP2AJCBLSQMOCEDERVTQLZHQF3Q, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Dec 27 04:27:09.957367413 (516h58m9.828450448s) [noise=true , address=77.90.31.12:4594 , piece_id=KOUSDJU2LDCXR33E5GHU4PDPYSP2AJCBLSQMOCEDERVTQLZHQF3Q, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957376080 (516h58m9.828459125s) [noise=true , address=64.49.13.169:4707 , piece_id=MCDCXRNL3UDPSMA7Z2OKFEIGE323WOO6NBLAHYOJ662ZQNJOAEFQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48910->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48910->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48910->64.49.13.169:4707: use of closed network connection Dec 27 04:27:09.957387151 (516h58m9.828470196s) [noise=true , address=64.49.13.169:4707 , piece_id=MCDCXRNL3UDPSMA7Z2OKFEIGE323WOO6NBLAHYOJ662ZQNJOAEFQ, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957643309 (516h58m9.828726354s) [noise=true , address=77.90.31.21:4891 , piece_id=XQV3KPGFJCOEYSWP5IYVS43PFNG4OXUS3EQA3MPO4HYBFA54YZMQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g): protocol: expected piece hash; context canceled; EOF Dec 27 04:27:09.957651324 (516h58m9.828734359s) [noise=true , address=77.90.31.21:4891 , piece_id=XQV3KPGFJCOEYSWP5IYVS43PFNG4OXUS3EQA3MPO4HYBFA54YZMQ, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957693745 (516h58m9.82877679s) [seg_pos={0 0}, upload=453579]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:27:09.957822199 (516h58m9.828905234s) [upload=453579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:27:09.957828862 (516h58m9.828911907s) [upload=453579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:27:10.257090208 (516h58m10.128173253s) [upload=453579]: Done uploading segments... Dec 27 04:27:10.257117090 (516h58m10.128200135s) [upload=453579]: Upload finished. err: <nil> Duration: 36.381631954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [........X.XX..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...] duration: 6.007118013s trace: https://jaeger.dev.storj.io/trace/ea8bcc5fd149f74 FAIL (2025-12-27 04:47:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 04:44:10.128811599 (517h15m9.999894644s) [upload=453831]: Starting upload Dec 27 04:44:10.128862866 (517h15m9.999945901s) [upload=453831]: Done starting upload Dec 27 04:44:10.128917270 (517h15m10.000000315s) [upload=453831]: Uploading segments... Dec 27 04:44:10.129178136 (517h15m10.000261181s) [upload=453831]: Got next segment. Inline: false Dec 27 04:44:10.129220737 (517h15m10.000303782s) [upload=453831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:44:10.129227892 (517h15m10.000310937s) [upload=453831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:44:10.368763939 (517h15m10.239846995s) [seg_pos={0 0}, upload=453831]: Begin upload segment... Dec 27 04:44:10.368847388 (517h15m10.239930433s) [seg_pos={0 0}, upload=453831]: Done begin upload segment. Dec 27 04:44:10.368884639 (517h15m10.239967674s) [noise=true , address=145.79.227.45:4229 , piece_id=5F3LTC2O33M6ZGCULAII7FHBWQB3AUMBWTQBFTAWJVWKHQVZZBIQ, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.368945204 (517h15m10.240028249s) [noise=true , address=145.79.227.46:4603 , piece_id=HUBIWBR5NA25OCY5KTPJ3EPPTYB43KSYVJAV2ILUNSX6466RTOBQ, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.369676968 (517h15m10.240760013s) [noise=true , address=145.79.227.49:4337 , piece_id=DE7AJ26VCD4MYZM3RG3IZCEQ2BQ7HDVCGAIM7SUFYJYE7ZAIZMOA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.369771167 (517h15m10.240854203s) [noise=true , address=145.79.227.6:4259 , piece_id=NDZ6A4HMGD4ABARPK2BA6WV3VLJ3PAY4UZ5FPIS757HGPCMS3M7A, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.369855267 (517h15m10.240938312s) [noise=true , address=145.79.227.2:4113 , piece_id=ZIYUWUY7STYY2QRZJBEABOJ76CTD5QDA5RUT4AY2QQMWU46LZFVQ, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.369909471 (517h15m10.240992516s) [noise=true , address=77.90.31.14:4661 , piece_id=U7Z6OSHZEGIAOM6YHFUYZA2HO4XPPICGLO66NOVDGSCPRIPXGYVA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.369966338 (517h15m10.241049384s) [noise=true , address=145.79.227.44:4194 , piece_id=R2FXZNQQVWZ64RPK3UQRFXP5K5ZQ6Q3H5DM5UWA6DKX3CZFG3NAA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.370008899 (517h15m10.241091934s) [noise=true , address=145.79.227.40:4050 , piece_id=MJXAFW4USW7UPCRCMRHO5B577MG46RQJO7KXLCPJZ5MPVW5KXZ2A, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.370047404 (517h15m10.241130459s) [noise=true , address=45.91.116.6:4976 , piece_id=7BX3Y53WHNFAYQKDG4VUPKMNROY4WI77VT6SMSOSBGYXGB6DZAEA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.370086187 (517h15m10.241169232s) [noise=true , address=77.90.31.15:4700 , piece_id=P3M6ALGGLPI7EFG4QZJYIUR2GH4G5WEAJHGL6JLXJOEGMYNVNEJA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.371136507 (517h15m10.242219552s) [noise=true , address=45.91.116.11:4119 , piece_id=ZPXHCS7WFEOSISDZFGDPS6JZ7GJSDPVG3ZJ6HATLZYNEJ72BTKXA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.371455194 (517h15m10.242538229s) [noise=true , address=145.79.227.16:4912 , piece_id=DNR4L24HDRCWGIPSLDWJKEYGQIKCJVU6HT4KH7E6JZDS7CLSGCVA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.371753081 (517h15m10.242836116s) [noise=true , address=145.79.227.35:4481 , piece_id=FGXRHLTPXRQBF7WUCOVNBR6UBAB2JZSSAKH5YARKHL2BVYXQH22Q, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.372098939 (517h15m10.243181974s) [noise=true , address=77.90.31.17:4770 , piece_id=E6BV73JKKAXTBYXZD6SZLAKKZ4B4AWTY44QRVGPX2WQM62XZKFMA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.372554167 (517h15m10.243637202s) [noise=true , address=145.79.227.33:4870 , piece_id=ITCISZEBGQNAQXPS23VIENDNOIEFT4MX5NEWGG55TLUCKVKFOAZQ, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.372985878 (517h15m10.244068924s) [noise=true , address=145.79.227.10:4403 , piece_id=BV3KZ5YQRTADNOEYCCKK4MZ5IKASLBPL5CGXQX72CJWB4T6WINNA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:12.247436734 (517h15m12.118519779s) [upload=453831]: Next returned nil segment Dec 27 04:44:12.247454417 (517h15m12.118537462s) [upload=453831]: Waiting for error group managing segments... Dec 27 04:44:12.544190383 (517h15m12.415273418s) [noise=true , address=77.90.31.15:4700 , piece_id=P3M6ALGGLPI7EFG4QZJYIUR2GH4G5WEAJHGL6JLXJOEGMYNVNEJA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:12.679144449 (517h15m12.550227484s) [noise=true , address=77.90.31.17:4770 , piece_id=E6BV73JKKAXTBYXZD6SZLAKKZ4B4AWTY44QRVGPX2WQM62XZKFMA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:12.961569875 (517h15m12.83265291s) [noise=true , address=77.90.31.14:4661 , piece_id=U7Z6OSHZEGIAOM6YHFUYZA2HO4XPPICGLO66NOVDGSCPRIPXGYVA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:12.962121255 (517h15m12.8332043s) [noise=true , address=45.91.116.6:4976 , piece_id=7BX3Y53WHNFAYQKDG4VUPKMNROY4WI77VT6SMSOSBGYXGB6DZAEA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:14.246144054 (517h15m14.117227089s) [noise=true , address=45.91.116.11:4119 , piece_id=ZPXHCS7WFEOSISDZFGDPS6JZ7GJSDPVG3ZJ6HATLZYNEJ72BTKXA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:14.315722553 (517h15m14.186805588s) [noise=true , address=145.79.227.45:4229 , piece_id=5F3LTC2O33M6ZGCULAII7FHBWQB3AUMBWTQBFTAWJVWKHQVZZBIQ, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:14.915580545 (517h15m14.78666359s) [noise=true , address=145.79.227.35:4481 , piece_id=FGXRHLTPXRQBF7WUCOVNBR6UBAB2JZSSAKH5YARKHL2BVYXQH22Q, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:15.315446127 (517h15m15.186529162s) [noise=true , address=145.79.227.49:4337 , piece_id=DE7AJ26VCD4MYZM3RG3IZCEQ2BQ7HDVCGAIM7SUFYJYE7ZAIZMOA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:15.334689009 (517h15m15.205772054s) [noise=true , address=145.79.227.33:4870 , piece_id=ITCISZEBGQNAQXPS23VIENDNOIEFT4MX5NEWGG55TLUCKVKFOAZQ, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:15.860890316 (517h15m15.731973361s) [noise=true , address=145.79.227.2:4113 , piece_id=ZIYUWUY7STYY2QRZJBEABOJ76CTD5QDA5RUT4AY2QQMWU46LZFVQ, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:16.118581865 (517h15m15.98966491s) [noise=true , address=145.79.227.44:4194 , piece_id=R2FXZNQQVWZ64RPK3UQRFXP5K5ZQ6Q3H5DM5UWA6DKX3CZFG3NAA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:16.232920486 (517h15m16.104003531s) [noise=true , address=145.79.227.40:4050 , piece_id=MJXAFW4USW7UPCRCMRHO5B577MG46RQJO7KXLCPJZ5MPVW5KXZ2A, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:16.760093934 (517h15m16.631176969s) [noise=true , address=145.79.227.46:4603 , piece_id=HUBIWBR5NA25OCY5KTPJ3EPPTYB43KSYVJAV2ILUNSX6466RTOBQ, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:16.760117078 (517h15m16.631200123s) [seg_pos={0 0}, upload=453831]: Segment reached optimal threshold of 13 pieces. Dec 27 04:44:16.760367755 (517h15m16.631450801s) [noise=true , address=145.79.227.16:4912 , piece_id=DNR4L24HDRCWGIPSLDWJKEYGQIKCJVU6HT4KH7E6JZDS7CLSGCVA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF Dec 27 04:44:16.760375961 (517h15m16.631459006s) [noise=true , address=145.79.227.16:4912 , piece_id=DNR4L24HDRCWGIPSLDWJKEYGQIKCJVU6HT4KH7E6JZDS7CLSGCVA, seg_pos={0 0}, upload=453831]: Long tail cancellation due to optimal threshold Dec 27 04:44:16.760432990 (517h15m16.631516025s) [noise=true , address=145.79.227.10:4403 , piece_id=BV3KZ5YQRTADNOEYCCKK4MZ5IKASLBPL5CGXQX72CJWB4T6WINNA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Dec 27 04:44:16.760442117 (517h15m16.631525152s) [noise=true , address=145.79.227.10:4403 , piece_id=BV3KZ5YQRTADNOEYCCKK4MZ5IKASLBPL5CGXQX72CJWB4T6WINNA, seg_pos={0 0}, upload=453831]: Long tail cancellation due to optimal threshold Dec 27 04:44:16.760660372 (517h15m16.631743407s) [noise=true , address=145.79.227.6:4259 , piece_id=NDZ6A4HMGD4ABARPK2BA6WV3VLJ3PAY4UZ5FPIS757HGPCMS3M7A, seg_pos={0 0}, upload=453831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: read tcp 188.166.225.170:44814->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:44:16.760667106 (517h15m16.631750141s) [noise=true , address=145.79.227.6:4259 , piece_id=NDZ6A4HMGD4ABARPK2BA6WV3VLJ3PAY4UZ5FPIS757HGPCMS3M7A, seg_pos={0 0}, upload=453831]: Long tail cancellation due to optimal threshold Dec 27 04:44:16.760712542 (517h15m16.631795587s) [seg_pos={0 0}, upload=453831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:44:16.760784679 (517h15m16.631867724s) [upload=453831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:44:16.760789238 (517h15m16.631872273s) [upload=453831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:44:17.059823651 (517h15m16.930906706s) [upload=453831]: Done uploading segments... Dec 27 04:44:17.059850502 (517h15m16.930933537s) [upload=453831]: Upload finished. err: <nil> Duration: 6.007147959s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.643167464s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:47:00 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.643195978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...................................X....................] duration: 49.37086972s trace: https://jaeger.dev.storj.io/trace/c87a834e8ef35c6 FAIL (2025-12-27 04:46:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, Piece ID: XUT6GYZRI6INV47H5JRO45PTZNNP63TJHONJWZ7ZX2C57K5BKGUQ) context deadline exceeded; piecestore: (Node ID: 1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz, Piece ID: 77VJGA5VAWH2SJWFAXIVGMGCWNO2KOCGTA45TKQDY23HEXBJSBQQ) context deadline exceeded; piecestore: (Node ID: 12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, Piece ID: J6ZYWC5MQE5UTGCDELO6OLH3SVEKFJUCEI5DNASQVLCPK6TNFSCQ) context deadline exceeded ======LOGS====== 27 04:27:06.503245535 (516h58m6.37432857s) [noise=true , address=45.91.116.7:4043 , piece_id=KC5UOUJS2Z3C5H3RF3RD4L6CSQEFBKZ6DBO24IKBDO43Z7B4KQQA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.503611160 (516h58m6.374694196s) [noise=true , address=77.90.31.15:4713 , piece_id=XUT6GYZRI6INV47H5JRO45PTZNNP63TJHONJWZ7ZX2C57K5BKGUQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.503968061 (516h58m6.375051106s) [noise=true , address=77.90.31.14:4678 , piece_id=PXSV7HHXMWBVPCJWXHDZTMPEVSGZSOOKRQE4NCNKJTXTDDCLSTAQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.504256129 (516h58m6.375339164s) [noise=true , address=64.49.13.169:4707 , piece_id=MCDCXRNL3UDPSMA7Z2OKFEIGE323WOO6NBLAHYOJ662ZQNJOAEFQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.504572461 (516h58m6.375655506s) [noise=true , address=145.79.227.46:4618 , piece_id=MCGD45OCXA2MSO5ESYDYJBY3WELG5IVFYVAEPKK4O2DDGXLN3BHQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.504944089 (516h58m6.376027124s) [noise=true , address=77.90.31.19:4862 , piece_id=77VJGA5VAWH2SJWFAXIVGMGCWNO2KOCGTA45TKQDY23HEXBJSBQQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.505755795 (516h58m6.37683883s) [noise=true , address=145.79.227.44:4226 , piece_id=BAVZROOB2AZAYV4BXQBMVIOQTYUSQOQIY4IXB6Z2YDXPATLRZD5Q, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.506138724 (516h58m6.377221759s) [noise=true , address=77.90.31.21:4891 , piece_id=XQV3KPGFJCOEYSWP5IYVS43PFNG4OXUS3EQA3MPO4HYBFA54YZMQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.506583922 (516h58m6.377666957s) [noise=true , address=45.91.116.11:4144 , piece_id=J6ZYWC5MQE5UTGCDELO6OLH3SVEKFJUCEI5DNASQVLCPK6TNFSCQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.506989984 (516h58m6.378073029s) [noise=true , address=145.79.227.40:4063 , piece_id=LNBR4YX5H6QCXVVKNGS7DHAPXUTPHG454PBQX5TN6F5S5BL3GHOA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.507382012 (516h58m6.378465057s) [noise=true , address=145.79.227.34:4918 , piece_id=6LSHYKYLMULRSTG6AQIQZESIG6CJXXCWX454N4GSTQSYR3MFJSBA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.507740744 (516h58m6.378823779s) [noise=true , address=145.79.227.30:4780 , piece_id=HMAX5PBRZU6Z7FAGCOYBNNVJ346FNMMCZXAAVUFUXVLAPQR7DQWA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.508083948 (516h58m6.379166993s) [noise=true , address=145.79.227.33:4890 , piece_id=EMRUMNQFNZUI7C5SPOBEXGCTZVOGZHMVL4YMMBBEX5KSN6HWOK2A, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.508422152 (516h58m6.379505187s) [noise=true , address=45.91.116.13:4163 , piece_id=NHNL5MZQPPINCZ5RGQADENPVBER22OUPWYV5T2M7JIA4OSHTGNXA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.508791535 (516h58m6.37987457s) [noise=true , address=77.90.31.12:4594 , piece_id=KOUSDJU2LDCXR33E5GHU4PDPYSP2AJCBLSQMOCEDERVTQLZHQF3Q, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.509145138 (516h58m6.380228183s) [noise=true , address=77.90.31.18:4812 , piece_id=XQTKFLX5AH7FFMBSW6PWMOMBVI4AWYYOXU6L25ANH6LWUDLWMC5Q, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:07.986779004 (516h58m7.85786204s) [upload=453579]: Next returned nil segment Dec 27 04:27:07.986792661 (516h58m7.857875696s) [upload=453579]: Waiting for error group managing segments... Dec 27 04:27:08.446303300 (516h58m8.317386335s) [noise=true , address=145.79.227.46:4618 , piece_id=MCGD45OCXA2MSO5ESYDYJBY3WELG5IVFYVAEPKK4O2DDGXLN3BHQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.543513967 (516h58m8.414597012s) [noise=true , address=77.90.31.19:4862 , piece_id=77VJGA5VAWH2SJWFAXIVGMGCWNO2KOCGTA45TKQDY23HEXBJSBQQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.572304228 (516h58m8.443387263s) [noise=true , address=77.90.31.16:4767 , piece_id=WBRSJ6RPJKEGFX7SN3GM3E2RS3OOFNGAWN4HCOQRSGYKXB62SWOA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.665325697 (516h58m8.536408742s) [noise=true , address=45.91.116.7:4043 , piece_id=KC5UOUJS2Z3C5H3RF3RD4L6CSQEFBKZ6DBO24IKBDO43Z7B4KQQA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.683431093 (516h58m8.554514138s) [noise=true , address=77.90.31.18:4812 , piece_id=XQTKFLX5AH7FFMBSW6PWMOMBVI4AWYYOXU6L25ANH6LWUDLWMC5Q, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.975168534 (516h58m8.846251579s) [noise=true , address=77.90.31.11:4563 , piece_id=LZKRVINZ2CVQ2IBNIPGRMLFRMHRQK7MANSL267MIQK26SDNIZCDQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.988860225 (516h58m8.85994326s) [noise=true , address=45.91.116.11:4144 , piece_id=J6ZYWC5MQE5UTGCDELO6OLH3SVEKFJUCEI5DNASQVLCPK6TNFSCQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.296067601 (516h58m9.167150646s) [noise=true , address=45.91.116.13:4163 , piece_id=NHNL5MZQPPINCZ5RGQADENPVBER22OUPWYV5T2M7JIA4OSHTGNXA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.300138683 (516h58m9.171221719s) [noise=true , address=145.79.227.34:4918 , piece_id=6LSHYKYLMULRSTG6AQIQZESIG6CJXXCWX454N4GSTQSYR3MFJSBA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.308910276 (516h58m9.179993321s) [noise=true , address=145.79.227.30:4780 , piece_id=HMAX5PBRZU6Z7FAGCOYBNNVJ346FNMMCZXAAVUFUXVLAPQR7DQWA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.338300491 (516h58m9.209383526s) [noise=true , address=145.79.227.44:4226 , piece_id=BAVZROOB2AZAYV4BXQBMVIOQTYUSQOQIY4IXB6Z2YDXPATLRZD5Q, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.425622296 (516h58m9.296705331s) [noise=true , address=145.79.227.33:4890 , piece_id=EMRUMNQFNZUI7C5SPOBEXGCTZVOGZHMVL4YMMBBEX5KSN6HWOK2A, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.868436633 (516h58m9.739519709s) [noise=true , address=77.90.31.15:4713 , piece_id=XUT6GYZRI6INV47H5JRO45PTZNNP63TJHONJWZ7ZX2C57K5BKGUQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.956938287 (516h58m9.828021342s) [noise=true , address=145.79.227.40:4063 , piece_id=LNBR4YX5H6QCXVVKNGS7DHAPXUTPHG454PBQX5TN6F5S5BL3GHOA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.956959947 (516h58m9.828042982s) [seg_pos={0 0}, upload=453579]: Segment reached optimal threshold of 14 pieces. Dec 27 04:27:09.957193332 (516h58m9.828276367s) [noise=true , address=77.90.31.14:4678 , piece_id=PXSV7HHXMWBVPCJWXHDZTMPEVSGZSOOKRQE4NCNKJTXTDDCLSTAQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: expected piece hash; context canceled; EOF Dec 27 04:27:09.957200034 (516h58m9.828283069s) [noise=true , address=77.90.31.14:4678 , piece_id=PXSV7HHXMWBVPCJWXHDZTMPEVSGZSOOKRQE4NCNKJTXTDDCLSTAQ, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957361282 (516h58m9.828444327s) [noise=true , address=77.90.31.12:4594 , piece_id=KOUSDJU2LDCXR33E5GHU4PDPYSP2AJCBLSQMOCEDERVTQLZHQF3Q, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Dec 27 04:27:09.957367413 (516h58m9.828450448s) [noise=true , address=77.90.31.12:4594 , piece_id=KOUSDJU2LDCXR33E5GHU4PDPYSP2AJCBLSQMOCEDERVTQLZHQF3Q, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957376080 (516h58m9.828459125s) [noise=true , address=64.49.13.169:4707 , piece_id=MCDCXRNL3UDPSMA7Z2OKFEIGE323WOO6NBLAHYOJ662ZQNJOAEFQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48910->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48910->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48910->64.49.13.169:4707: use of closed network connection Dec 27 04:27:09.957387151 (516h58m9.828470196s) [noise=true , address=64.49.13.169:4707 , piece_id=MCDCXRNL3UDPSMA7Z2OKFEIGE323WOO6NBLAHYOJ662ZQNJOAEFQ, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957643309 (516h58m9.828726354s) [noise=true , address=77.90.31.21:4891 , piece_id=XQV3KPGFJCOEYSWP5IYVS43PFNG4OXUS3EQA3MPO4HYBFA54YZMQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g): protocol: expected piece hash; context canceled; EOF Dec 27 04:27:09.957651324 (516h58m9.828734359s) [noise=true , address=77.90.31.21:4891 , piece_id=XQV3KPGFJCOEYSWP5IYVS43PFNG4OXUS3EQA3MPO4HYBFA54YZMQ, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957693745 (516h58m9.82877679s) [seg_pos={0 0}, upload=453579]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:27:09.957822199 (516h58m9.828905234s) [upload=453579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:27:09.957828862 (516h58m9.828911907s) [upload=453579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:27:10.257090208 (516h58m10.128173253s) [upload=453579]: Done uploading segments... Dec 27 04:27:10.257117090 (516h58m10.128200135s) [upload=453579]: Upload finished. err: <nil> Duration: 49.370919334s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.491500223s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:46:42 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.491577741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X........X.XX..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX..] duration: 5.766536813s trace: https://jaeger.dev.storj.io/trace/ea8bcc5fd149f74 FAIL (2025-12-27 04:46:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 04:44:10.128811599 (517h15m9.999894644s) [upload=453831]: Starting upload Dec 27 04:44:10.128862866 (517h15m9.999945901s) [upload=453831]: Done starting upload Dec 27 04:44:10.128917270 (517h15m10.000000315s) [upload=453831]: Uploading segments... Dec 27 04:44:10.129178136 (517h15m10.000261181s) [upload=453831]: Got next segment. Inline: false Dec 27 04:44:10.129220737 (517h15m10.000303782s) [upload=453831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:44:10.129227892 (517h15m10.000310937s) [upload=453831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:44:10.368763939 (517h15m10.239846995s) [seg_pos={0 0}, upload=453831]: Begin upload segment... Dec 27 04:44:10.368847388 (517h15m10.239930433s) [seg_pos={0 0}, upload=453831]: Done begin upload segment. Dec 27 04:44:10.368884639 (517h15m10.239967674s) [noise=true , address=145.79.227.45:4229 , piece_id=5F3LTC2O33M6ZGCULAII7FHBWQB3AUMBWTQBFTAWJVWKHQVZZBIQ, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.368945204 (517h15m10.240028249s) [noise=true , address=145.79.227.46:4603 , piece_id=HUBIWBR5NA25OCY5KTPJ3EPPTYB43KSYVJAV2ILUNSX6466RTOBQ, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.369676968 (517h15m10.240760013s) [noise=true , address=145.79.227.49:4337 , piece_id=DE7AJ26VCD4MYZM3RG3IZCEQ2BQ7HDVCGAIM7SUFYJYE7ZAIZMOA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.369771167 (517h15m10.240854203s) [noise=true , address=145.79.227.6:4259 , piece_id=NDZ6A4HMGD4ABARPK2BA6WV3VLJ3PAY4UZ5FPIS757HGPCMS3M7A, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.369855267 (517h15m10.240938312s) [noise=true , address=145.79.227.2:4113 , piece_id=ZIYUWUY7STYY2QRZJBEABOJ76CTD5QDA5RUT4AY2QQMWU46LZFVQ, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.369909471 (517h15m10.240992516s) [noise=true , address=77.90.31.14:4661 , piece_id=U7Z6OSHZEGIAOM6YHFUYZA2HO4XPPICGLO66NOVDGSCPRIPXGYVA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.369966338 (517h15m10.241049384s) [noise=true , address=145.79.227.44:4194 , piece_id=R2FXZNQQVWZ64RPK3UQRFXP5K5ZQ6Q3H5DM5UWA6DKX3CZFG3NAA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.370008899 (517h15m10.241091934s) [noise=true , address=145.79.227.40:4050 , piece_id=MJXAFW4USW7UPCRCMRHO5B577MG46RQJO7KXLCPJZ5MPVW5KXZ2A, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.370047404 (517h15m10.241130459s) [noise=true , address=45.91.116.6:4976 , piece_id=7BX3Y53WHNFAYQKDG4VUPKMNROY4WI77VT6SMSOSBGYXGB6DZAEA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.370086187 (517h15m10.241169232s) [noise=true , address=77.90.31.15:4700 , piece_id=P3M6ALGGLPI7EFG4QZJYIUR2GH4G5WEAJHGL6JLXJOEGMYNVNEJA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.371136507 (517h15m10.242219552s) [noise=true , address=45.91.116.11:4119 , piece_id=ZPXHCS7WFEOSISDZFGDPS6JZ7GJSDPVG3ZJ6HATLZYNEJ72BTKXA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.371455194 (517h15m10.242538229s) [noise=true , address=145.79.227.16:4912 , piece_id=DNR4L24HDRCWGIPSLDWJKEYGQIKCJVU6HT4KH7E6JZDS7CLSGCVA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.371753081 (517h15m10.242836116s) [noise=true , address=145.79.227.35:4481 , piece_id=FGXRHLTPXRQBF7WUCOVNBR6UBAB2JZSSAKH5YARKHL2BVYXQH22Q, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.372098939 (517h15m10.243181974s) [noise=true , address=77.90.31.17:4770 , piece_id=E6BV73JKKAXTBYXZD6SZLAKKZ4B4AWTY44QRVGPX2WQM62XZKFMA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.372554167 (517h15m10.243637202s) [noise=true , address=145.79.227.33:4870 , piece_id=ITCISZEBGQNAQXPS23VIENDNOIEFT4MX5NEWGG55TLUCKVKFOAZQ, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:10.372985878 (517h15m10.244068924s) [noise=true , address=145.79.227.10:4403 , piece_id=BV3KZ5YQRTADNOEYCCKK4MZ5IKASLBPL5CGXQX72CJWB4T6WINNA, seg_pos={0 0}, upload=453831]: Uploading piece... Dec 27 04:44:12.247436734 (517h15m12.118519779s) [upload=453831]: Next returned nil segment Dec 27 04:44:12.247454417 (517h15m12.118537462s) [upload=453831]: Waiting for error group managing segments... Dec 27 04:44:12.544190383 (517h15m12.415273418s) [noise=true , address=77.90.31.15:4700 , piece_id=P3M6ALGGLPI7EFG4QZJYIUR2GH4G5WEAJHGL6JLXJOEGMYNVNEJA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:12.679144449 (517h15m12.550227484s) [noise=true , address=77.90.31.17:4770 , piece_id=E6BV73JKKAXTBYXZD6SZLAKKZ4B4AWTY44QRVGPX2WQM62XZKFMA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:12.961569875 (517h15m12.83265291s) [noise=true , address=77.90.31.14:4661 , piece_id=U7Z6OSHZEGIAOM6YHFUYZA2HO4XPPICGLO66NOVDGSCPRIPXGYVA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:12.962121255 (517h15m12.8332043s) [noise=true , address=45.91.116.6:4976 , piece_id=7BX3Y53WHNFAYQKDG4VUPKMNROY4WI77VT6SMSOSBGYXGB6DZAEA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:14.246144054 (517h15m14.117227089s) [noise=true , address=45.91.116.11:4119 , piece_id=ZPXHCS7WFEOSISDZFGDPS6JZ7GJSDPVG3ZJ6HATLZYNEJ72BTKXA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:14.315722553 (517h15m14.186805588s) [noise=true , address=145.79.227.45:4229 , piece_id=5F3LTC2O33M6ZGCULAII7FHBWQB3AUMBWTQBFTAWJVWKHQVZZBIQ, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:14.915580545 (517h15m14.78666359s) [noise=true , address=145.79.227.35:4481 , piece_id=FGXRHLTPXRQBF7WUCOVNBR6UBAB2JZSSAKH5YARKHL2BVYXQH22Q, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:15.315446127 (517h15m15.186529162s) [noise=true , address=145.79.227.49:4337 , piece_id=DE7AJ26VCD4MYZM3RG3IZCEQ2BQ7HDVCGAIM7SUFYJYE7ZAIZMOA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:15.334689009 (517h15m15.205772054s) [noise=true , address=145.79.227.33:4870 , piece_id=ITCISZEBGQNAQXPS23VIENDNOIEFT4MX5NEWGG55TLUCKVKFOAZQ, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:15.860890316 (517h15m15.731973361s) [noise=true , address=145.79.227.2:4113 , piece_id=ZIYUWUY7STYY2QRZJBEABOJ76CTD5QDA5RUT4AY2QQMWU46LZFVQ, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:16.118581865 (517h15m15.98966491s) [noise=true , address=145.79.227.44:4194 , piece_id=R2FXZNQQVWZ64RPK3UQRFXP5K5ZQ6Q3H5DM5UWA6DKX3CZFG3NAA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:16.232920486 (517h15m16.104003531s) [noise=true , address=145.79.227.40:4050 , piece_id=MJXAFW4USW7UPCRCMRHO5B577MG46RQJO7KXLCPJZ5MPVW5KXZ2A, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:16.760093934 (517h15m16.631176969s) [noise=true , address=145.79.227.46:4603 , piece_id=HUBIWBR5NA25OCY5KTPJ3EPPTYB43KSYVJAV2ILUNSX6466RTOBQ, seg_pos={0 0}, upload=453831]: Done uploading piece. err: <nil> Dec 27 04:44:16.760117078 (517h15m16.631200123s) [seg_pos={0 0}, upload=453831]: Segment reached optimal threshold of 13 pieces. Dec 27 04:44:16.760367755 (517h15m16.631450801s) [noise=true , address=145.79.227.16:4912 , piece_id=DNR4L24HDRCWGIPSLDWJKEYGQIKCJVU6HT4KH7E6JZDS7CLSGCVA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF Dec 27 04:44:16.760375961 (517h15m16.631459006s) [noise=true , address=145.79.227.16:4912 , piece_id=DNR4L24HDRCWGIPSLDWJKEYGQIKCJVU6HT4KH7E6JZDS7CLSGCVA, seg_pos={0 0}, upload=453831]: Long tail cancellation due to optimal threshold Dec 27 04:44:16.760432990 (517h15m16.631516025s) [noise=true , address=145.79.227.10:4403 , piece_id=BV3KZ5YQRTADNOEYCCKK4MZ5IKASLBPL5CGXQX72CJWB4T6WINNA, seg_pos={0 0}, upload=453831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Dec 27 04:44:16.760442117 (517h15m16.631525152s) [noise=true , address=145.79.227.10:4403 , piece_id=BV3KZ5YQRTADNOEYCCKK4MZ5IKASLBPL5CGXQX72CJWB4T6WINNA, seg_pos={0 0}, upload=453831]: Long tail cancellation due to optimal threshold Dec 27 04:44:16.760660372 (517h15m16.631743407s) [noise=true , address=145.79.227.6:4259 , piece_id=NDZ6A4HMGD4ABARPK2BA6WV3VLJ3PAY4UZ5FPIS757HGPCMS3M7A, seg_pos={0 0}, upload=453831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: read tcp 188.166.225.170:44814->145.79.227.6:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:44:16.760667106 (517h15m16.631750141s) [noise=true , address=145.79.227.6:4259 , piece_id=NDZ6A4HMGD4ABARPK2BA6WV3VLJ3PAY4UZ5FPIS757HGPCMS3M7A, seg_pos={0 0}, upload=453831]: Long tail cancellation due to optimal threshold Dec 27 04:44:16.760712542 (517h15m16.631795587s) [seg_pos={0 0}, upload=453831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:44:16.760784679 (517h15m16.631867724s) [upload=453831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:44:16.760789238 (517h15m16.631872273s) [upload=453831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:44:17.059823651 (517h15m16.930906706s) [upload=453831]: Done uploading segments... Dec 27 04:44:17.059850502 (517h15m16.930933537s) [upload=453831]: Upload finished. err: <nil> Duration: 5.766562851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...................................X...................] duration: 52.350138419s trace: https://jaeger.dev.storj.io/trace/c87a834e8ef35c6 FAIL (2025-12-27 04:46:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, Piece ID: XUT6GYZRI6INV47H5JRO45PTZNNP63TJHONJWZ7ZX2C57K5BKGUQ) context deadline exceeded; piecestore: (Node ID: 1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz, Piece ID: 77VJGA5VAWH2SJWFAXIVGMGCWNO2KOCGTA45TKQDY23HEXBJSBQQ) context deadline exceeded; piecestore: (Node ID: 12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, Piece ID: J6ZYWC5MQE5UTGCDELO6OLH3SVEKFJUCEI5DNASQVLCPK6TNFSCQ) context deadline exceeded ======LOGS====== 27 04:27:06.503245535 (516h58m6.37432857s) [noise=true , address=45.91.116.7:4043 , piece_id=KC5UOUJS2Z3C5H3RF3RD4L6CSQEFBKZ6DBO24IKBDO43Z7B4KQQA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.503611160 (516h58m6.374694196s) [noise=true , address=77.90.31.15:4713 , piece_id=XUT6GYZRI6INV47H5JRO45PTZNNP63TJHONJWZ7ZX2C57K5BKGUQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.503968061 (516h58m6.375051106s) [noise=true , address=77.90.31.14:4678 , piece_id=PXSV7HHXMWBVPCJWXHDZTMPEVSGZSOOKRQE4NCNKJTXTDDCLSTAQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.504256129 (516h58m6.375339164s) [noise=true , address=64.49.13.169:4707 , piece_id=MCDCXRNL3UDPSMA7Z2OKFEIGE323WOO6NBLAHYOJ662ZQNJOAEFQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.504572461 (516h58m6.375655506s) [noise=true , address=145.79.227.46:4618 , piece_id=MCGD45OCXA2MSO5ESYDYJBY3WELG5IVFYVAEPKK4O2DDGXLN3BHQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.504944089 (516h58m6.376027124s) [noise=true , address=77.90.31.19:4862 , piece_id=77VJGA5VAWH2SJWFAXIVGMGCWNO2KOCGTA45TKQDY23HEXBJSBQQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.505755795 (516h58m6.37683883s) [noise=true , address=145.79.227.44:4226 , piece_id=BAVZROOB2AZAYV4BXQBMVIOQTYUSQOQIY4IXB6Z2YDXPATLRZD5Q, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.506138724 (516h58m6.377221759s) [noise=true , address=77.90.31.21:4891 , piece_id=XQV3KPGFJCOEYSWP5IYVS43PFNG4OXUS3EQA3MPO4HYBFA54YZMQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.506583922 (516h58m6.377666957s) [noise=true , address=45.91.116.11:4144 , piece_id=J6ZYWC5MQE5UTGCDELO6OLH3SVEKFJUCEI5DNASQVLCPK6TNFSCQ, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.506989984 (516h58m6.378073029s) [noise=true , address=145.79.227.40:4063 , piece_id=LNBR4YX5H6QCXVVKNGS7DHAPXUTPHG454PBQX5TN6F5S5BL3GHOA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.507382012 (516h58m6.378465057s) [noise=true , address=145.79.227.34:4918 , piece_id=6LSHYKYLMULRSTG6AQIQZESIG6CJXXCWX454N4GSTQSYR3MFJSBA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.507740744 (516h58m6.378823779s) [noise=true , address=145.79.227.30:4780 , piece_id=HMAX5PBRZU6Z7FAGCOYBNNVJ346FNMMCZXAAVUFUXVLAPQR7DQWA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.508083948 (516h58m6.379166993s) [noise=true , address=145.79.227.33:4890 , piece_id=EMRUMNQFNZUI7C5SPOBEXGCTZVOGZHMVL4YMMBBEX5KSN6HWOK2A, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.508422152 (516h58m6.379505187s) [noise=true , address=45.91.116.13:4163 , piece_id=NHNL5MZQPPINCZ5RGQADENPVBER22OUPWYV5T2M7JIA4OSHTGNXA, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.508791535 (516h58m6.37987457s) [noise=true , address=77.90.31.12:4594 , piece_id=KOUSDJU2LDCXR33E5GHU4PDPYSP2AJCBLSQMOCEDERVTQLZHQF3Q, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:06.509145138 (516h58m6.380228183s) [noise=true , address=77.90.31.18:4812 , piece_id=XQTKFLX5AH7FFMBSW6PWMOMBVI4AWYYOXU6L25ANH6LWUDLWMC5Q, seg_pos={0 0}, upload=453579]: Uploading piece... Dec 27 04:27:07.986779004 (516h58m7.85786204s) [upload=453579]: Next returned nil segment Dec 27 04:27:07.986792661 (516h58m7.857875696s) [upload=453579]: Waiting for error group managing segments... Dec 27 04:27:08.446303300 (516h58m8.317386335s) [noise=true , address=145.79.227.46:4618 , piece_id=MCGD45OCXA2MSO5ESYDYJBY3WELG5IVFYVAEPKK4O2DDGXLN3BHQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.543513967 (516h58m8.414597012s) [noise=true , address=77.90.31.19:4862 , piece_id=77VJGA5VAWH2SJWFAXIVGMGCWNO2KOCGTA45TKQDY23HEXBJSBQQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.572304228 (516h58m8.443387263s) [noise=true , address=77.90.31.16:4767 , piece_id=WBRSJ6RPJKEGFX7SN3GM3E2RS3OOFNGAWN4HCOQRSGYKXB62SWOA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.665325697 (516h58m8.536408742s) [noise=true , address=45.91.116.7:4043 , piece_id=KC5UOUJS2Z3C5H3RF3RD4L6CSQEFBKZ6DBO24IKBDO43Z7B4KQQA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.683431093 (516h58m8.554514138s) [noise=true , address=77.90.31.18:4812 , piece_id=XQTKFLX5AH7FFMBSW6PWMOMBVI4AWYYOXU6L25ANH6LWUDLWMC5Q, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.975168534 (516h58m8.846251579s) [noise=true , address=77.90.31.11:4563 , piece_id=LZKRVINZ2CVQ2IBNIPGRMLFRMHRQK7MANSL267MIQK26SDNIZCDQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:08.988860225 (516h58m8.85994326s) [noise=true , address=45.91.116.11:4144 , piece_id=J6ZYWC5MQE5UTGCDELO6OLH3SVEKFJUCEI5DNASQVLCPK6TNFSCQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.296067601 (516h58m9.167150646s) [noise=true , address=45.91.116.13:4163 , piece_id=NHNL5MZQPPINCZ5RGQADENPVBER22OUPWYV5T2M7JIA4OSHTGNXA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.300138683 (516h58m9.171221719s) [noise=true , address=145.79.227.34:4918 , piece_id=6LSHYKYLMULRSTG6AQIQZESIG6CJXXCWX454N4GSTQSYR3MFJSBA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.308910276 (516h58m9.179993321s) [noise=true , address=145.79.227.30:4780 , piece_id=HMAX5PBRZU6Z7FAGCOYBNNVJ346FNMMCZXAAVUFUXVLAPQR7DQWA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.338300491 (516h58m9.209383526s) [noise=true , address=145.79.227.44:4226 , piece_id=BAVZROOB2AZAYV4BXQBMVIOQTYUSQOQIY4IXB6Z2YDXPATLRZD5Q, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.425622296 (516h58m9.296705331s) [noise=true , address=145.79.227.33:4890 , piece_id=EMRUMNQFNZUI7C5SPOBEXGCTZVOGZHMVL4YMMBBEX5KSN6HWOK2A, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.868436633 (516h58m9.739519709s) [noise=true , address=77.90.31.15:4713 , piece_id=XUT6GYZRI6INV47H5JRO45PTZNNP63TJHONJWZ7ZX2C57K5BKGUQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.956938287 (516h58m9.828021342s) [noise=true , address=145.79.227.40:4063 , piece_id=LNBR4YX5H6QCXVVKNGS7DHAPXUTPHG454PBQX5TN6F5S5BL3GHOA, seg_pos={0 0}, upload=453579]: Done uploading piece. err: <nil> Dec 27 04:27:09.956959947 (516h58m9.828042982s) [seg_pos={0 0}, upload=453579]: Segment reached optimal threshold of 14 pieces. Dec 27 04:27:09.957193332 (516h58m9.828276367s) [noise=true , address=77.90.31.14:4678 , piece_id=PXSV7HHXMWBVPCJWXHDZTMPEVSGZSOOKRQE4NCNKJTXTDDCLSTAQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: expected piece hash; context canceled; EOF Dec 27 04:27:09.957200034 (516h58m9.828283069s) [noise=true , address=77.90.31.14:4678 , piece_id=PXSV7HHXMWBVPCJWXHDZTMPEVSGZSOOKRQE4NCNKJTXTDDCLSTAQ, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957361282 (516h58m9.828444327s) [noise=true , address=77.90.31.12:4594 , piece_id=KOUSDJU2LDCXR33E5GHU4PDPYSP2AJCBLSQMOCEDERVTQLZHQF3Q, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Dec 27 04:27:09.957367413 (516h58m9.828450448s) [noise=true , address=77.90.31.12:4594 , piece_id=KOUSDJU2LDCXR33E5GHU4PDPYSP2AJCBLSQMOCEDERVTQLZHQF3Q, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957376080 (516h58m9.828459125s) [noise=true , address=64.49.13.169:4707 , piece_id=MCDCXRNL3UDPSMA7Z2OKFEIGE323WOO6NBLAHYOJ662ZQNJOAEFQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:48910->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:48910->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48910->64.49.13.169:4707: use of closed network connection Dec 27 04:27:09.957387151 (516h58m9.828470196s) [noise=true , address=64.49.13.169:4707 , piece_id=MCDCXRNL3UDPSMA7Z2OKFEIGE323WOO6NBLAHYOJ662ZQNJOAEFQ, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957643309 (516h58m9.828726354s) [noise=true , address=77.90.31.21:4891 , piece_id=XQV3KPGFJCOEYSWP5IYVS43PFNG4OXUS3EQA3MPO4HYBFA54YZMQ, seg_pos={0 0}, upload=453579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17RDAerfx8AW5NsgWLTXLNDbuAEWQFT3SVM4iwdmuBzWrML83g): protocol: expected piece hash; context canceled; EOF Dec 27 04:27:09.957651324 (516h58m9.828734359s) [noise=true , address=77.90.31.21:4891 , piece_id=XQV3KPGFJCOEYSWP5IYVS43PFNG4OXUS3EQA3MPO4HYBFA54YZMQ, seg_pos={0 0}, upload=453579]: Long tail cancellation due to optimal threshold Dec 27 04:27:09.957693745 (516h58m9.82877679s) [seg_pos={0 0}, upload=453579]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:27:09.957822199 (516h58m9.828905234s) [upload=453579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:27:09.957828862 (516h58m9.828911907s) [upload=453579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:27:10.257090208 (516h58m10.128173253s) [upload=453579]: Done uploading segments... Dec 27 04:27:10.257117090 (516h58m10.128200135s) [upload=453579]: Upload finished. err: <nil> Duration: 52.350167174s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)