Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X.......X................................................] duration: 4.987454985s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:36:30 UTC) orbiter warning: context deadline exceeded Duration: 4.987481816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X......X...........................X............] duration: 19.983289382s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:36:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 19.983310003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X...X..XX..X.] duration: 5.990286985s trace: https://jaeger.dev.storj.io/trace/6caffa4e2ce5e163 FAIL (2025-12-29 07:36:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:34:56.500546503 (568h5m56.371629548s) [upload=498056]: Starting upload Dec 29 07:34:56.500610265 (568h5m56.3716933s) [upload=498056]: Done starting upload Dec 29 07:34:56.500682863 (568h5m56.371765898s) [upload=498056]: Uploading segments... Dec 29 07:34:56.500830334 (568h5m56.371913379s) [upload=498056]: Got next segment. Inline: false Dec 29 07:34:56.500872714 (568h5m56.371955749s) [upload=498056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:34:56.500880439 (568h5m56.371963484s) [upload=498056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:34:56.748873194 (568h5m56.619956259s) [seg_pos={0 0}, upload=498056]: Begin upload segment... Dec 29 07:34:56.748945461 (568h5m56.620028506s) [seg_pos={0 0}, upload=498056]: Done begin upload segment. Dec 29 07:34:56.748993823 (568h5m56.620076858s) [noise=true , address=145.79.227.45:4231 , piece_id=KBFLM7JUNJADASCGQ5KUNRHFJA2BDQD75RQO2GGLDUX23IVTFR6Q, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.749024832 (568h5m56.620107897s) [noise=true , address=145.79.227.41:4086 , piece_id=NMTZWLACUKXTWC2FZS65N36ASK2MMSWRVALULM35BSBM5KDFM37A, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.749728442 (568h5m56.620811497s) [noise=true , address=145.79.227.37:4566 , piece_id=O3EHH4RTDQT5HRJBHFOYIPZTTIH6NUJO4GZKHNWAFFIPRUQZI2OQ, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.750308306 (568h5m56.621391351s) [noise=true , address=145.79.227.4:4186 , piece_id=LWOD6U7CD3KZOCALLGJ73QZANAQAA3RRIXRN64SW4WNIY7L2YJ7Q, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.750818278 (568h5m56.621901323s) [noise=true , address=45.91.116.13:4154 , piece_id=CFZSLR6UXTRHKQFFP6LJBIPGGQQB2T3IBZUDTUMFUXFBMB5ZSZDQ, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.751380037 (568h5m56.622463092s) [noise=true , address=45.91.116.11:4117 , piece_id=M4V2U6M7FXRWVHSHZVHHTMM3RXLWJ4I5NDG5CQUTITKZXBHFMLQQ, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.751794116 (568h5m56.622877161s) [noise=true , address=145.79.227.15:4874 , piece_id=KWN5ILBMDUDZD3H4TWAYWWRUZSJ24DZYF6XRFIMBFKGAXEO4Z4IQ, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.753376458 (568h5m56.624459604s) [noise=true , address=45.91.116.6:4975 , piece_id=OMFXDYB62HFLBRQGMDI4YGU5NEDHXXYMD4VSHJ2QZSA2R257FEGA, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.753970980 (568h5m56.625054025s) [noise=true , address=145.79.227.35:4482 , piece_id=F3BDZDB6MU4B3SPUVFAQYFKBDNYJHEAVB32BHTT3WBN3DI2JQAYQ, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.754549482 (568h5m56.625632527s) [noise=true , address=145.79.227.29:4941 , piece_id=WDT4JQBLQQPREEBKX7BBFH744CNEEHMP43Z3OZHCTGT35ZCIHH4A, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.754943672 (568h5m56.626026718s) [noise=true , address=145.79.227.49:4337 , piece_id=J4L4JHCIWLMYFSZZE3VC37M67Z5VQUWM6SWPGWAWITJ7VSN4JUOA, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.755405061 (568h5m56.626488096s) [noise=true , address=45.91.116.9:4263 , piece_id=E5YP4HTBD2XIL3P4DKAJCLDI5KRTJZXPM3NCJWBY362SRGXAKFHA, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.755926283 (568h5m56.627009318s) [noise=true , address=145.79.227.9:4367 , piece_id=B3ADELC6IO3VEAR5LZCYQ5ZD5EHXUM6R3EQYC7P6AIN33NIJO4EA, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.756344961 (568h5m56.627427996s) [noise=true , address=145.79.227.3:4150 , piece_id=EZB3DWIKICKWIM7VS72Y7K66SNLTUMDDXZMBH35ODGOXQTWK4SDA, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.756697932 (568h5m56.627780978s) [noise=true , address=145.79.227.26:4653 , piece_id=RDLL4TE4MBHUN5NERKOXX3KFY5INMBLRRW754YER6U5Q7FL6HNIA, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.757250545 (568h5m56.62833358s) [noise=true , address=45.91.116.7:4011 , piece_id=ZAOSY3LGPS3SPNMVSD3PHXKUDFP66JK4DZ7S35F6VVQRC25GCNMQ, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:58.433243033 (568h5m58.304326068s) [upload=498056]: Next returned nil segment Dec 29 07:34:58.433267239 (568h5m58.304350274s) [upload=498056]: Waiting for error group managing segments... Dec 29 07:34:58.741882828 (568h5m58.612965863s) [noise=true , address=145.79.227.4:4186 , piece_id=LWOD6U7CD3KZOCALLGJ73QZANAQAA3RRIXRN64SW4WNIY7L2YJ7Q, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.742171569 (568h5m58.613254624s) [noise=true , address=145.79.227.9:4367 , piece_id=B3ADELC6IO3VEAR5LZCYQ5ZD5EHXUM6R3EQYC7P6AIN33NIJO4EA, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.744136330 (568h5m58.615219375s) [noise=true , address=145.79.227.26:4653 , piece_id=RDLL4TE4MBHUN5NERKOXX3KFY5INMBLRRW754YER6U5Q7FL6HNIA, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.757946095 (568h5m58.62902913s) [noise=true , address=145.79.227.37:4566 , piece_id=O3EHH4RTDQT5HRJBHFOYIPZTTIH6NUJO4GZKHNWAFFIPRUQZI2OQ, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.757979970 (568h5m58.629063005s) [noise=true , address=145.79.227.15:4874 , piece_id=KWN5ILBMDUDZD3H4TWAYWWRUZSJ24DZYF6XRFIMBFKGAXEO4Z4IQ, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.761159855 (568h5m58.63224289s) [noise=true , address=145.79.227.3:4150 , piece_id=EZB3DWIKICKWIM7VS72Y7K66SNLTUMDDXZMBH35ODGOXQTWK4SDA, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.761188118 (568h5m58.632271153s) [noise=true , address=145.79.227.35:4482 , piece_id=F3BDZDB6MU4B3SPUVFAQYFKBDNYJHEAVB32BHTT3WBN3DI2JQAYQ, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.765003454 (568h5m58.636086489s) [noise=true , address=145.79.227.45:4231 , piece_id=KBFLM7JUNJADASCGQ5KUNRHFJA2BDQD75RQO2GGLDUX23IVTFR6Q, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.765205268 (568h5m58.636288313s) [noise=true , address=145.79.227.41:4086 , piece_id=NMTZWLACUKXTWC2FZS65N36ASK2MMSWRVALULM35BSBM5KDFM37A, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.868138498 (568h5m58.739221543s) [noise=true , address=145.79.227.29:4941 , piece_id=WDT4JQBLQQPREEBKX7BBFH744CNEEHMP43Z3OZHCTGT35ZCIHH4A, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.999603067 (568h5m58.870686102s) [noise=true , address=45.91.116.9:4263 , piece_id=E5YP4HTBD2XIL3P4DKAJCLDI5KRTJZXPM3NCJWBY362SRGXAKFHA, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:59.122881737 (568h5m58.993964782s) [noise=true , address=45.91.116.7:4011 , piece_id=ZAOSY3LGPS3SPNMVSD3PHXKUDFP66JK4DZ7S35F6VVQRC25GCNMQ, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:59.129390982 (568h5m59.000474027s) [noise=true , address=45.91.116.13:4154 , piece_id=CFZSLR6UXTRHKQFFP6LJBIPGGQQB2T3IBZUDTUMFUXFBMB5ZSZDQ, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:59.129417072 (568h5m59.000500117s) [seg_pos={0 0}, upload=498056]: Segment reached optimal threshold of 13 pieces. Dec 29 07:34:59.129657119 (568h5m59.000740154s) [noise=true , address=145.79.227.49:4337 , piece_id=J4L4JHCIWLMYFSZZE3VC37M67Z5VQUWM6SWPGWAWITJ7VSN4JUOA, seg_pos={0 0}, upload=498056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 07:34:59.129664202 (568h5m59.000747247s) [noise=true , address=145.79.227.49:4337 , piece_id=J4L4JHCIWLMYFSZZE3VC37M67Z5VQUWM6SWPGWAWITJ7VSN4JUOA, seg_pos={0 0}, upload=498056]: Long tail cancellation due to optimal threshold Dec 29 07:34:59.129843313 (568h5m59.000926358s) [noise=true , address=45.91.116.6:4975 , piece_id=OMFXDYB62HFLBRQGMDI4YGU5NEDHXXYMD4VSHJ2QZSA2R257FEGA, seg_pos={0 0}, upload=498056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 07:34:59.129859163 (568h5m59.000942198s) [noise=true , address=45.91.116.6:4975 , piece_id=OMFXDYB62HFLBRQGMDI4YGU5NEDHXXYMD4VSHJ2QZSA2R257FEGA, seg_pos={0 0}, upload=498056]: Long tail cancellation due to optimal threshold Dec 29 07:34:59.130068281 (568h5m59.001151316s) [noise=true , address=45.91.116.11:4117 , piece_id=M4V2U6M7FXRWVHSHZVHHTMM3RXLWJ4I5NDG5CQUTITKZXBHFMLQQ, seg_pos={0 0}, upload=498056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: context canceled; context canceled Dec 29 07:34:59.130074845 (568h5m59.00115788s) [noise=true , address=45.91.116.11:4117 , piece_id=M4V2U6M7FXRWVHSHZVHHTMM3RXLWJ4I5NDG5CQUTITKZXBHFMLQQ, seg_pos={0 0}, upload=498056]: Long tail cancellation due to optimal threshold Dec 29 07:34:59.130090374 (568h5m59.001173409s) [seg_pos={0 0}, upload=498056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:34:59.130163443 (568h5m59.001246488s) [upload=498056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:34:59.130171458 (568h5m59.001254523s) [upload=498056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:34:59.419279684 (568h5m59.290362729s) [upload=498056]: Done uploading segments... Dec 29 07:34:59.419303067 (568h5m59.290386112s) [upload=498056]: Upload finished. err: <nil> Duration: 5.990316722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X......X...........................X...........] duration: 36.49685993s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:35:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 36.496872584s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X.......X...............................................] duration: 3.363651772s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:35:28 UTC) orbiter warning: context deadline exceeded Duration: 3.363676949s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X...X..XX..X] duration: 30.001960044s trace: https://jaeger.dev.storj.io/trace/6caffa4e2ce5e163 FAIL (2025-12-29 07:35:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:34:56.500546503 (568h5m56.371629548s) [upload=498056]: Starting upload Dec 29 07:34:56.500610265 (568h5m56.3716933s) [upload=498056]: Done starting upload Dec 29 07:34:56.500682863 (568h5m56.371765898s) [upload=498056]: Uploading segments... Dec 29 07:34:56.500830334 (568h5m56.371913379s) [upload=498056]: Got next segment. Inline: false Dec 29 07:34:56.500872714 (568h5m56.371955749s) [upload=498056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:34:56.500880439 (568h5m56.371963484s) [upload=498056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:34:56.748873194 (568h5m56.619956259s) [seg_pos={0 0}, upload=498056]: Begin upload segment... Dec 29 07:34:56.748945461 (568h5m56.620028506s) [seg_pos={0 0}, upload=498056]: Done begin upload segment. Dec 29 07:34:56.748993823 (568h5m56.620076858s) [noise=true , address=145.79.227.45:4231 , piece_id=KBFLM7JUNJADASCGQ5KUNRHFJA2BDQD75RQO2GGLDUX23IVTFR6Q, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.749024832 (568h5m56.620107897s) [noise=true , address=145.79.227.41:4086 , piece_id=NMTZWLACUKXTWC2FZS65N36ASK2MMSWRVALULM35BSBM5KDFM37A, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.749728442 (568h5m56.620811497s) [noise=true , address=145.79.227.37:4566 , piece_id=O3EHH4RTDQT5HRJBHFOYIPZTTIH6NUJO4GZKHNWAFFIPRUQZI2OQ, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.750308306 (568h5m56.621391351s) [noise=true , address=145.79.227.4:4186 , piece_id=LWOD6U7CD3KZOCALLGJ73QZANAQAA3RRIXRN64SW4WNIY7L2YJ7Q, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.750818278 (568h5m56.621901323s) [noise=true , address=45.91.116.13:4154 , piece_id=CFZSLR6UXTRHKQFFP6LJBIPGGQQB2T3IBZUDTUMFUXFBMB5ZSZDQ, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.751380037 (568h5m56.622463092s) [noise=true , address=45.91.116.11:4117 , piece_id=M4V2U6M7FXRWVHSHZVHHTMM3RXLWJ4I5NDG5CQUTITKZXBHFMLQQ, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.751794116 (568h5m56.622877161s) [noise=true , address=145.79.227.15:4874 , piece_id=KWN5ILBMDUDZD3H4TWAYWWRUZSJ24DZYF6XRFIMBFKGAXEO4Z4IQ, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.753376458 (568h5m56.624459604s) [noise=true , address=45.91.116.6:4975 , piece_id=OMFXDYB62HFLBRQGMDI4YGU5NEDHXXYMD4VSHJ2QZSA2R257FEGA, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.753970980 (568h5m56.625054025s) [noise=true , address=145.79.227.35:4482 , piece_id=F3BDZDB6MU4B3SPUVFAQYFKBDNYJHEAVB32BHTT3WBN3DI2JQAYQ, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.754549482 (568h5m56.625632527s) [noise=true , address=145.79.227.29:4941 , piece_id=WDT4JQBLQQPREEBKX7BBFH744CNEEHMP43Z3OZHCTGT35ZCIHH4A, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.754943672 (568h5m56.626026718s) [noise=true , address=145.79.227.49:4337 , piece_id=J4L4JHCIWLMYFSZZE3VC37M67Z5VQUWM6SWPGWAWITJ7VSN4JUOA, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.755405061 (568h5m56.626488096s) [noise=true , address=45.91.116.9:4263 , piece_id=E5YP4HTBD2XIL3P4DKAJCLDI5KRTJZXPM3NCJWBY362SRGXAKFHA, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.755926283 (568h5m56.627009318s) [noise=true , address=145.79.227.9:4367 , piece_id=B3ADELC6IO3VEAR5LZCYQ5ZD5EHXUM6R3EQYC7P6AIN33NIJO4EA, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.756344961 (568h5m56.627427996s) [noise=true , address=145.79.227.3:4150 , piece_id=EZB3DWIKICKWIM7VS72Y7K66SNLTUMDDXZMBH35ODGOXQTWK4SDA, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.756697932 (568h5m56.627780978s) [noise=true , address=145.79.227.26:4653 , piece_id=RDLL4TE4MBHUN5NERKOXX3KFY5INMBLRRW754YER6U5Q7FL6HNIA, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:56.757250545 (568h5m56.62833358s) [noise=true , address=45.91.116.7:4011 , piece_id=ZAOSY3LGPS3SPNMVSD3PHXKUDFP66JK4DZ7S35F6VVQRC25GCNMQ, seg_pos={0 0}, upload=498056]: Uploading piece... Dec 29 07:34:58.433243033 (568h5m58.304326068s) [upload=498056]: Next returned nil segment Dec 29 07:34:58.433267239 (568h5m58.304350274s) [upload=498056]: Waiting for error group managing segments... Dec 29 07:34:58.741882828 (568h5m58.612965863s) [noise=true , address=145.79.227.4:4186 , piece_id=LWOD6U7CD3KZOCALLGJ73QZANAQAA3RRIXRN64SW4WNIY7L2YJ7Q, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.742171569 (568h5m58.613254624s) [noise=true , address=145.79.227.9:4367 , piece_id=B3ADELC6IO3VEAR5LZCYQ5ZD5EHXUM6R3EQYC7P6AIN33NIJO4EA, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.744136330 (568h5m58.615219375s) [noise=true , address=145.79.227.26:4653 , piece_id=RDLL4TE4MBHUN5NERKOXX3KFY5INMBLRRW754YER6U5Q7FL6HNIA, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.757946095 (568h5m58.62902913s) [noise=true , address=145.79.227.37:4566 , piece_id=O3EHH4RTDQT5HRJBHFOYIPZTTIH6NUJO4GZKHNWAFFIPRUQZI2OQ, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.757979970 (568h5m58.629063005s) [noise=true , address=145.79.227.15:4874 , piece_id=KWN5ILBMDUDZD3H4TWAYWWRUZSJ24DZYF6XRFIMBFKGAXEO4Z4IQ, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.761159855 (568h5m58.63224289s) [noise=true , address=145.79.227.3:4150 , piece_id=EZB3DWIKICKWIM7VS72Y7K66SNLTUMDDXZMBH35ODGOXQTWK4SDA, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.761188118 (568h5m58.632271153s) [noise=true , address=145.79.227.35:4482 , piece_id=F3BDZDB6MU4B3SPUVFAQYFKBDNYJHEAVB32BHTT3WBN3DI2JQAYQ, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.765003454 (568h5m58.636086489s) [noise=true , address=145.79.227.45:4231 , piece_id=KBFLM7JUNJADASCGQ5KUNRHFJA2BDQD75RQO2GGLDUX23IVTFR6Q, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.765205268 (568h5m58.636288313s) [noise=true , address=145.79.227.41:4086 , piece_id=NMTZWLACUKXTWC2FZS65N36ASK2MMSWRVALULM35BSBM5KDFM37A, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.868138498 (568h5m58.739221543s) [noise=true , address=145.79.227.29:4941 , piece_id=WDT4JQBLQQPREEBKX7BBFH744CNEEHMP43Z3OZHCTGT35ZCIHH4A, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:58.999603067 (568h5m58.870686102s) [noise=true , address=45.91.116.9:4263 , piece_id=E5YP4HTBD2XIL3P4DKAJCLDI5KRTJZXPM3NCJWBY362SRGXAKFHA, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:59.122881737 (568h5m58.993964782s) [noise=true , address=45.91.116.7:4011 , piece_id=ZAOSY3LGPS3SPNMVSD3PHXKUDFP66JK4DZ7S35F6VVQRC25GCNMQ, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:59.129390982 (568h5m59.000474027s) [noise=true , address=45.91.116.13:4154 , piece_id=CFZSLR6UXTRHKQFFP6LJBIPGGQQB2T3IBZUDTUMFUXFBMB5ZSZDQ, seg_pos={0 0}, upload=498056]: Done uploading piece. err: <nil> Dec 29 07:34:59.129417072 (568h5m59.000500117s) [seg_pos={0 0}, upload=498056]: Segment reached optimal threshold of 13 pieces. Dec 29 07:34:59.129657119 (568h5m59.000740154s) [noise=true , address=145.79.227.49:4337 , piece_id=J4L4JHCIWLMYFSZZE3VC37M67Z5VQUWM6SWPGWAWITJ7VSN4JUOA, seg_pos={0 0}, upload=498056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 07:34:59.129664202 (568h5m59.000747247s) [noise=true , address=145.79.227.49:4337 , piece_id=J4L4JHCIWLMYFSZZE3VC37M67Z5VQUWM6SWPGWAWITJ7VSN4JUOA, seg_pos={0 0}, upload=498056]: Long tail cancellation due to optimal threshold Dec 29 07:34:59.129843313 (568h5m59.000926358s) [noise=true , address=45.91.116.6:4975 , piece_id=OMFXDYB62HFLBRQGMDI4YGU5NEDHXXYMD4VSHJ2QZSA2R257FEGA, seg_pos={0 0}, upload=498056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 07:34:59.129859163 (568h5m59.000942198s) [noise=true , address=45.91.116.6:4975 , piece_id=OMFXDYB62HFLBRQGMDI4YGU5NEDHXXYMD4VSHJ2QZSA2R257FEGA, seg_pos={0 0}, upload=498056]: Long tail cancellation due to optimal threshold Dec 29 07:34:59.130068281 (568h5m59.001151316s) [noise=true , address=45.91.116.11:4117 , piece_id=M4V2U6M7FXRWVHSHZVHHTMM3RXLWJ4I5NDG5CQUTITKZXBHFMLQQ, seg_pos={0 0}, upload=498056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: context canceled; context canceled Dec 29 07:34:59.130074845 (568h5m59.00115788s) [noise=true , address=45.91.116.11:4117 , piece_id=M4V2U6M7FXRWVHSHZVHHTMM3RXLWJ4I5NDG5CQUTITKZXBHFMLQQ, seg_pos={0 0}, upload=498056]: Long tail cancellation due to optimal threshold Dec 29 07:34:59.130090374 (568h5m59.001173409s) [seg_pos={0 0}, upload=498056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:34:59.130163443 (568h5m59.001246488s) [upload=498056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:34:59.130171458 (568h5m59.001254523s) [upload=498056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:34:59.419279684 (568h5m59.290362729s) [upload=498056]: Done uploading segments... Dec 29 07:34:59.419303067 (568h5m59.290386112s) [upload=498056]: Upload finished. err: <nil> Duration: 30.001976956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X......X...........................X..........] duration: 30.951686167s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:35:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 30.951717335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X......X...........................X.........] duration: 15.421922918s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:34:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 15.421953948s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X.......X..............................................] duration: 3.477054949s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:34:19 UTC) orbiter warning: context deadline exceeded Duration: 3.477074836s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X.......X.............................................] duration: 4.648206036s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:33:58 UTC) orbiter warning: context deadline exceeded Duration: 4.648227126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X......X...........................X........] duration: 33.327976632s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:33:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 33.327997985s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X...X..XX..] duration: 5.99789205s trace: https://jaeger.dev.storj.io/trace/766344e680c257fb FAIL (2025-12-29 07:33:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: A5TU3EG7EKWTKABQVWQZQE4EX7UJ5MWB3XYQ3WEEOBU5TTUXPJSA) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: XOSBBKNWP6VGWESKQFFT3572S4MNBPN5I4Z2R33C6S44LEPG4Y4Q) context deadline exceeded ======LOGS====== Dec 29 07:32:13.569110368 (568h3m13.440193413s) [upload=498021]: Starting upload Dec 29 07:32:13.569202323 (568h3m13.440285388s) [upload=498021]: Done starting upload Dec 29 07:32:13.569276785 (568h3m13.44035983s) [upload=498021]: Uploading segments... Dec 29 07:32:13.569888139 (568h3m13.440971194s) [upload=498021]: Got next segment. Inline: false Dec 29 07:32:13.569931692 (568h3m13.441014727s) [upload=498021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:32:13.569938425 (568h3m13.44102146s) [upload=498021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:32:13.817380025 (568h3m13.68846307s) [seg_pos={0 0}, upload=498021]: Begin upload segment... Dec 29 07:32:13.817440170 (568h3m13.688523205s) [seg_pos={0 0}, upload=498021]: Done begin upload segment. Dec 29 07:32:13.817475257 (568h3m13.688558302s) [noise=true , address=145.79.227.42:4122 , piece_id=TJJS7WYFIVVVAJ2OO7GXO6VPDNU2N3EYLDW3UDA6ZEWKNBBMUQLA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.817616185 (568h3m13.68869922s) [noise=true , address=145.79.227.47:4266 , piece_id=C6BUKWRREDCSERVR4UPGSMKRXCTOQ6ZT24VQCSR54J5UKDOBUUJQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.818441877 (568h3m13.689524913s) [noise=true , address=77.90.31.19:4841 , piece_id=X3YPQ6QLWSOHCPAZQDEA3OGCYDGIIOIUAYYWJOW62637T2WOG7KA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.819042501 (568h3m13.690125546s) [noise=true , address=145.79.227.14:4838 , piece_id=62OG3EOAY4YMLFVSYHU6OF7Q46MJFVS5ZO6FOVVAO2GAKL45DWWQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.819544186 (568h3m13.690627221s) [noise=true , address=45.91.116.8:4046 , piece_id=UKLRKQMJJ3RDNCQ7E3TWZL3MPLFAHWA6CMLZPPTFJJNXIMGPQ5KQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.820027075 (568h3m13.691110131s) [noise=true , address=145.79.227.48:4301 , piece_id=3CYFNU64ELINNXLA2WUDNYEGGILOY6WYMU365KE3VXDAOE7VIN4Q, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.820479427 (568h3m13.691562462s) [noise=true , address=45.91.116.13:4153 , piece_id=IUMYOKZMJUVP2LUSY7S5TTNQBCF6LAOGKP2U6NBXMM2ZN3TWSLLQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.820801921 (568h3m13.691884966s) [noise=true , address=145.79.227.52:4445 , piece_id=RTCV457EQY6BQQGUBKDJSHXJPPGDNMGYAIFEQH4SRTPVHR7ENEXA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.822044126 (568h3m13.693127171s) [noise=true , address=45.91.116.14:4192 , piece_id=A5TU3EG7EKWTKABQVWQZQE4EX7UJ5MWB3XYQ3WEEOBU5TTUXPJSA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.822265227 (568h3m13.693348272s) [noise=true , address=45.91.116.9:4263 , piece_id=D7JFAXL7CDMK3Y2V6O6ZFO4B3FN4I4LO2CSKKDT7XC3XVSZZQKKQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.823803146 (568h3m13.694886181s) [noise=true , address=77.90.31.13:4626 , piece_id=XOSBBKNWP6VGWESKQFFT3572S4MNBPN5I4Z2R33C6S44LEPG4Y4Q, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.825561574 (568h3m13.696644619s) [noise=true , address=45.91.116.10:4083 , piece_id=DM7FQALVYY6B3OXSEOWORDJMYWFQQRI7PBGYXZHS3OINTYWFWKJA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.829174975 (568h3m13.70025801s) [noise=true , address=145.79.227.7:4295 , piece_id=Z7TC3PGCCD26G42HFKKIOBT7VXTLKOAWF44WGT7JBA2D2IZQIFHA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.833459964 (568h3m13.704543009s) [noise=true , address=45.91.116.7:4010 , piece_id=JI3LOKGGWOJOYFNIRFUFVYBHCBVG6BFJHVLQ7EF6F447OGU6SUZA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.834279094 (568h3m13.705362149s) [noise=true , address=45.91.116.12:4225 , piece_id=DYTULOUH7OMZZWSHHLBAMMCE7AGYIWLLKFX22NNL4HONFAPPO6RQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.837406649 (568h3m13.708489684s) [noise=true , address=45.91.116.6:4974 , piece_id=4TN43QOOIJGNNBCI6ELIQYUKIYEWQ6YLIL572AXHJZHCXOJTT47A, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:15.538987291 (568h3m15.410070336s) [upload=498021]: Next returned nil segment Dec 29 07:32:15.539002550 (568h3m15.410085595s) [upload=498021]: Waiting for error group managing segments... Dec 29 07:32:15.822958259 (568h3m15.694041305s) [noise=true , address=145.79.227.47:4266 , piece_id=C6BUKWRREDCSERVR4UPGSMKRXCTOQ6ZT24VQCSR54J5UKDOBUUJQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.823745880 (568h3m15.694828925s) [noise=true , address=145.79.227.42:4122 , piece_id=TJJS7WYFIVVVAJ2OO7GXO6VPDNU2N3EYLDW3UDA6ZEWKNBBMUQLA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.826646713 (568h3m15.697729748s) [noise=true , address=145.79.227.14:4838 , piece_id=62OG3EOAY4YMLFVSYHU6OF7Q46MJFVS5ZO6FOVVAO2GAKL45DWWQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.829780210 (568h3m15.700863245s) [noise=true , address=145.79.227.48:4301 , piece_id=3CYFNU64ELINNXLA2WUDNYEGGILOY6WYMU365KE3VXDAOE7VIN4Q, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.830236929 (568h3m15.701319964s) [noise=true , address=145.79.227.7:4295 , piece_id=Z7TC3PGCCD26G42HFKKIOBT7VXTLKOAWF44WGT7JBA2D2IZQIFHA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.839131236 (568h3m15.710214271s) [noise=true , address=145.79.227.52:4445 , piece_id=RTCV457EQY6BQQGUBKDJSHXJPPGDNMGYAIFEQH4SRTPVHR7ENEXA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.071461683 (568h3m15.942544728s) [noise=true , address=45.91.116.7:4010 , piece_id=JI3LOKGGWOJOYFNIRFUFVYBHCBVG6BFJHVLQ7EF6F447OGU6SUZA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.155399806 (568h3m16.026482851s) [noise=true , address=77.90.31.13:4626 , piece_id=XOSBBKNWP6VGWESKQFFT3572S4MNBPN5I4Z2R33C6S44LEPG4Y4Q, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.184915247 (568h3m16.055998292s) [noise=true , address=45.91.116.14:4192 , piece_id=A5TU3EG7EKWTKABQVWQZQE4EX7UJ5MWB3XYQ3WEEOBU5TTUXPJSA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.190219708 (568h3m16.061302754s) [noise=true , address=45.91.116.8:4046 , piece_id=UKLRKQMJJ3RDNCQ7E3TWZL3MPLFAHWA6CMLZPPTFJJNXIMGPQ5KQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.220678457 (568h3m16.091761492s) [noise=true , address=45.91.116.6:4974 , piece_id=4TN43QOOIJGNNBCI6ELIQYUKIYEWQ6YLIL572AXHJZHCXOJTT47A, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.273564812 (568h3m16.144647847s) [noise=true , address=45.91.116.10:4083 , piece_id=DM7FQALVYY6B3OXSEOWORDJMYWFQQRI7PBGYXZHS3OINTYWFWKJA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.757690240 (568h3m16.628773285s) [noise=true , address=77.90.31.19:4841 , piece_id=X3YPQ6QLWSOHCPAZQDEA3OGCYDGIIOIUAYYWJOW62637T2WOG7KA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.757720176 (568h3m16.628803211s) [seg_pos={0 0}, upload=498021]: Segment reached optimal threshold of 13 pieces. Dec 29 07:32:16.758190502 (568h3m16.629273547s) [noise=true , address=45.91.116.12:4225 , piece_id=DYTULOUH7OMZZWSHHLBAMMCE7AGYIWLLKFX22NNL4HONFAPPO6RQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 07:32:16.758206211 (568h3m16.629289246s) [noise=true , address=45.91.116.12:4225 , piece_id=DYTULOUH7OMZZWSHHLBAMMCE7AGYIWLLKFX22NNL4HONFAPPO6RQ, seg_pos={0 0}, upload=498021]: Long tail cancellation due to optimal threshold Dec 29 07:32:16.758288419 (568h3m16.629371454s) [noise=true , address=45.91.116.13:4153 , piece_id=IUMYOKZMJUVP2LUSY7S5TTNQBCF6LAOGKP2U6NBXMM2ZN3TWSLLQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:45698->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:45698->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45698->45.91.116.13:4153: use of closed network connection Dec 29 07:32:16.758298758 (568h3m16.629381803s) [noise=true , address=45.91.116.13:4153 , piece_id=IUMYOKZMJUVP2LUSY7S5TTNQBCF6LAOGKP2U6NBXMM2ZN3TWSLLQ, seg_pos={0 0}, upload=498021]: Long tail cancellation due to optimal threshold Dec 29 07:32:16.758441339 (568h3m16.629524374s) [noise=true , address=45.91.116.9:4263 , piece_id=D7JFAXL7CDMK3Y2V6O6ZFO4B3FN4I4LO2CSKKDT7XC3XVSZZQKKQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 29 07:32:16.758450486 (568h3m16.629533522s) [noise=true , address=45.91.116.9:4263 , piece_id=D7JFAXL7CDMK3Y2V6O6ZFO4B3FN4I4LO2CSKKDT7XC3XVSZZQKKQ, seg_pos={0 0}, upload=498021]: Long tail cancellation due to optimal threshold Dec 29 07:32:16.758473270 (568h3m16.629556305s) [seg_pos={0 0}, upload=498021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:32:16.758640359 (568h3m16.629723394s) [upload=498021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:32:16.758647021 (568h3m16.629730066s) [upload=498021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:32:17.054367334 (568h3m16.925450379s) [upload=498021]: Done uploading segments... Dec 29 07:32:17.054392401 (568h3m16.925475456s) [upload=498021]: Upload finished. err: <nil> Duration: 5.997920715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X......X...........................X.......] duration: 53.055130517s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:33:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 53.055152671s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X.......X............................................] duration: 3.859009907s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:32:58 UTC) orbiter warning: context deadline exceeded Duration: 3.859034624s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X...X..XX.] duration: 5.96027947s trace: https://jaeger.dev.storj.io/trace/766344e680c257fb FAIL (2025-12-29 07:32:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: A5TU3EG7EKWTKABQVWQZQE4EX7UJ5MWB3XYQ3WEEOBU5TTUXPJSA) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: XOSBBKNWP6VGWESKQFFT3572S4MNBPN5I4Z2R33C6S44LEPG4Y4Q) context deadline exceeded ======LOGS====== Dec 29 07:32:13.569110368 (568h3m13.440193413s) [upload=498021]: Starting upload Dec 29 07:32:13.569202323 (568h3m13.440285388s) [upload=498021]: Done starting upload Dec 29 07:32:13.569276785 (568h3m13.44035983s) [upload=498021]: Uploading segments... Dec 29 07:32:13.569888139 (568h3m13.440971194s) [upload=498021]: Got next segment. Inline: false Dec 29 07:32:13.569931692 (568h3m13.441014727s) [upload=498021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:32:13.569938425 (568h3m13.44102146s) [upload=498021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:32:13.817380025 (568h3m13.68846307s) [seg_pos={0 0}, upload=498021]: Begin upload segment... Dec 29 07:32:13.817440170 (568h3m13.688523205s) [seg_pos={0 0}, upload=498021]: Done begin upload segment. Dec 29 07:32:13.817475257 (568h3m13.688558302s) [noise=true , address=145.79.227.42:4122 , piece_id=TJJS7WYFIVVVAJ2OO7GXO6VPDNU2N3EYLDW3UDA6ZEWKNBBMUQLA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.817616185 (568h3m13.68869922s) [noise=true , address=145.79.227.47:4266 , piece_id=C6BUKWRREDCSERVR4UPGSMKRXCTOQ6ZT24VQCSR54J5UKDOBUUJQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.818441877 (568h3m13.689524913s) [noise=true , address=77.90.31.19:4841 , piece_id=X3YPQ6QLWSOHCPAZQDEA3OGCYDGIIOIUAYYWJOW62637T2WOG7KA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.819042501 (568h3m13.690125546s) [noise=true , address=145.79.227.14:4838 , piece_id=62OG3EOAY4YMLFVSYHU6OF7Q46MJFVS5ZO6FOVVAO2GAKL45DWWQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.819544186 (568h3m13.690627221s) [noise=true , address=45.91.116.8:4046 , piece_id=UKLRKQMJJ3RDNCQ7E3TWZL3MPLFAHWA6CMLZPPTFJJNXIMGPQ5KQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.820027075 (568h3m13.691110131s) [noise=true , address=145.79.227.48:4301 , piece_id=3CYFNU64ELINNXLA2WUDNYEGGILOY6WYMU365KE3VXDAOE7VIN4Q, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.820479427 (568h3m13.691562462s) [noise=true , address=45.91.116.13:4153 , piece_id=IUMYOKZMJUVP2LUSY7S5TTNQBCF6LAOGKP2U6NBXMM2ZN3TWSLLQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.820801921 (568h3m13.691884966s) [noise=true , address=145.79.227.52:4445 , piece_id=RTCV457EQY6BQQGUBKDJSHXJPPGDNMGYAIFEQH4SRTPVHR7ENEXA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.822044126 (568h3m13.693127171s) [noise=true , address=45.91.116.14:4192 , piece_id=A5TU3EG7EKWTKABQVWQZQE4EX7UJ5MWB3XYQ3WEEOBU5TTUXPJSA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.822265227 (568h3m13.693348272s) [noise=true , address=45.91.116.9:4263 , piece_id=D7JFAXL7CDMK3Y2V6O6ZFO4B3FN4I4LO2CSKKDT7XC3XVSZZQKKQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.823803146 (568h3m13.694886181s) [noise=true , address=77.90.31.13:4626 , piece_id=XOSBBKNWP6VGWESKQFFT3572S4MNBPN5I4Z2R33C6S44LEPG4Y4Q, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.825561574 (568h3m13.696644619s) [noise=true , address=45.91.116.10:4083 , piece_id=DM7FQALVYY6B3OXSEOWORDJMYWFQQRI7PBGYXZHS3OINTYWFWKJA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.829174975 (568h3m13.70025801s) [noise=true , address=145.79.227.7:4295 , piece_id=Z7TC3PGCCD26G42HFKKIOBT7VXTLKOAWF44WGT7JBA2D2IZQIFHA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.833459964 (568h3m13.704543009s) [noise=true , address=45.91.116.7:4010 , piece_id=JI3LOKGGWOJOYFNIRFUFVYBHCBVG6BFJHVLQ7EF6F447OGU6SUZA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.834279094 (568h3m13.705362149s) [noise=true , address=45.91.116.12:4225 , piece_id=DYTULOUH7OMZZWSHHLBAMMCE7AGYIWLLKFX22NNL4HONFAPPO6RQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.837406649 (568h3m13.708489684s) [noise=true , address=45.91.116.6:4974 , piece_id=4TN43QOOIJGNNBCI6ELIQYUKIYEWQ6YLIL572AXHJZHCXOJTT47A, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:15.538987291 (568h3m15.410070336s) [upload=498021]: Next returned nil segment Dec 29 07:32:15.539002550 (568h3m15.410085595s) [upload=498021]: Waiting for error group managing segments... Dec 29 07:32:15.822958259 (568h3m15.694041305s) [noise=true , address=145.79.227.47:4266 , piece_id=C6BUKWRREDCSERVR4UPGSMKRXCTOQ6ZT24VQCSR54J5UKDOBUUJQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.823745880 (568h3m15.694828925s) [noise=true , address=145.79.227.42:4122 , piece_id=TJJS7WYFIVVVAJ2OO7GXO6VPDNU2N3EYLDW3UDA6ZEWKNBBMUQLA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.826646713 (568h3m15.697729748s) [noise=true , address=145.79.227.14:4838 , piece_id=62OG3EOAY4YMLFVSYHU6OF7Q46MJFVS5ZO6FOVVAO2GAKL45DWWQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.829780210 (568h3m15.700863245s) [noise=true , address=145.79.227.48:4301 , piece_id=3CYFNU64ELINNXLA2WUDNYEGGILOY6WYMU365KE3VXDAOE7VIN4Q, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.830236929 (568h3m15.701319964s) [noise=true , address=145.79.227.7:4295 , piece_id=Z7TC3PGCCD26G42HFKKIOBT7VXTLKOAWF44WGT7JBA2D2IZQIFHA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.839131236 (568h3m15.710214271s) [noise=true , address=145.79.227.52:4445 , piece_id=RTCV457EQY6BQQGUBKDJSHXJPPGDNMGYAIFEQH4SRTPVHR7ENEXA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.071461683 (568h3m15.942544728s) [noise=true , address=45.91.116.7:4010 , piece_id=JI3LOKGGWOJOYFNIRFUFVYBHCBVG6BFJHVLQ7EF6F447OGU6SUZA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.155399806 (568h3m16.026482851s) [noise=true , address=77.90.31.13:4626 , piece_id=XOSBBKNWP6VGWESKQFFT3572S4MNBPN5I4Z2R33C6S44LEPG4Y4Q, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.184915247 (568h3m16.055998292s) [noise=true , address=45.91.116.14:4192 , piece_id=A5TU3EG7EKWTKABQVWQZQE4EX7UJ5MWB3XYQ3WEEOBU5TTUXPJSA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.190219708 (568h3m16.061302754s) [noise=true , address=45.91.116.8:4046 , piece_id=UKLRKQMJJ3RDNCQ7E3TWZL3MPLFAHWA6CMLZPPTFJJNXIMGPQ5KQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.220678457 (568h3m16.091761492s) [noise=true , address=45.91.116.6:4974 , piece_id=4TN43QOOIJGNNBCI6ELIQYUKIYEWQ6YLIL572AXHJZHCXOJTT47A, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.273564812 (568h3m16.144647847s) [noise=true , address=45.91.116.10:4083 , piece_id=DM7FQALVYY6B3OXSEOWORDJMYWFQQRI7PBGYXZHS3OINTYWFWKJA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.757690240 (568h3m16.628773285s) [noise=true , address=77.90.31.19:4841 , piece_id=X3YPQ6QLWSOHCPAZQDEA3OGCYDGIIOIUAYYWJOW62637T2WOG7KA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.757720176 (568h3m16.628803211s) [seg_pos={0 0}, upload=498021]: Segment reached optimal threshold of 13 pieces. Dec 29 07:32:16.758190502 (568h3m16.629273547s) [noise=true , address=45.91.116.12:4225 , piece_id=DYTULOUH7OMZZWSHHLBAMMCE7AGYIWLLKFX22NNL4HONFAPPO6RQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 07:32:16.758206211 (568h3m16.629289246s) [noise=true , address=45.91.116.12:4225 , piece_id=DYTULOUH7OMZZWSHHLBAMMCE7AGYIWLLKFX22NNL4HONFAPPO6RQ, seg_pos={0 0}, upload=498021]: Long tail cancellation due to optimal threshold Dec 29 07:32:16.758288419 (568h3m16.629371454s) [noise=true , address=45.91.116.13:4153 , piece_id=IUMYOKZMJUVP2LUSY7S5TTNQBCF6LAOGKP2U6NBXMM2ZN3TWSLLQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:45698->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:45698->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45698->45.91.116.13:4153: use of closed network connection Dec 29 07:32:16.758298758 (568h3m16.629381803s) [noise=true , address=45.91.116.13:4153 , piece_id=IUMYOKZMJUVP2LUSY7S5TTNQBCF6LAOGKP2U6NBXMM2ZN3TWSLLQ, seg_pos={0 0}, upload=498021]: Long tail cancellation due to optimal threshold Dec 29 07:32:16.758441339 (568h3m16.629524374s) [noise=true , address=45.91.116.9:4263 , piece_id=D7JFAXL7CDMK3Y2V6O6ZFO4B3FN4I4LO2CSKKDT7XC3XVSZZQKKQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 29 07:32:16.758450486 (568h3m16.629533522s) [noise=true , address=45.91.116.9:4263 , piece_id=D7JFAXL7CDMK3Y2V6O6ZFO4B3FN4I4LO2CSKKDT7XC3XVSZZQKKQ, seg_pos={0 0}, upload=498021]: Long tail cancellation due to optimal threshold Dec 29 07:32:16.758473270 (568h3m16.629556305s) [seg_pos={0 0}, upload=498021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:32:16.758640359 (568h3m16.629723394s) [upload=498021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:32:16.758647021 (568h3m16.629730066s) [upload=498021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:32:17.054367334 (568h3m16.925450379s) [upload=498021]: Done uploading segments... Dec 29 07:32:17.054392401 (568h3m16.925475456s) [upload=498021]: Upload finished. err: <nil> Duration: 5.960322722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXX.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X...X..XX] duration: 30.003783942s trace: https://jaeger.dev.storj.io/trace/766344e680c257fb FAIL (2025-12-29 07:32:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: A5TU3EG7EKWTKABQVWQZQE4EX7UJ5MWB3XYQ3WEEOBU5TTUXPJSA) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: XOSBBKNWP6VGWESKQFFT3572S4MNBPN5I4Z2R33C6S44LEPG4Y4Q) context deadline exceeded ======LOGS====== Dec 29 07:32:13.569110368 (568h3m13.440193413s) [upload=498021]: Starting upload Dec 29 07:32:13.569202323 (568h3m13.440285388s) [upload=498021]: Done starting upload Dec 29 07:32:13.569276785 (568h3m13.44035983s) [upload=498021]: Uploading segments... Dec 29 07:32:13.569888139 (568h3m13.440971194s) [upload=498021]: Got next segment. Inline: false Dec 29 07:32:13.569931692 (568h3m13.441014727s) [upload=498021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:32:13.569938425 (568h3m13.44102146s) [upload=498021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:32:13.817380025 (568h3m13.68846307s) [seg_pos={0 0}, upload=498021]: Begin upload segment... Dec 29 07:32:13.817440170 (568h3m13.688523205s) [seg_pos={0 0}, upload=498021]: Done begin upload segment. Dec 29 07:32:13.817475257 (568h3m13.688558302s) [noise=true , address=145.79.227.42:4122 , piece_id=TJJS7WYFIVVVAJ2OO7GXO6VPDNU2N3EYLDW3UDA6ZEWKNBBMUQLA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.817616185 (568h3m13.68869922s) [noise=true , address=145.79.227.47:4266 , piece_id=C6BUKWRREDCSERVR4UPGSMKRXCTOQ6ZT24VQCSR54J5UKDOBUUJQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.818441877 (568h3m13.689524913s) [noise=true , address=77.90.31.19:4841 , piece_id=X3YPQ6QLWSOHCPAZQDEA3OGCYDGIIOIUAYYWJOW62637T2WOG7KA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.819042501 (568h3m13.690125546s) [noise=true , address=145.79.227.14:4838 , piece_id=62OG3EOAY4YMLFVSYHU6OF7Q46MJFVS5ZO6FOVVAO2GAKL45DWWQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.819544186 (568h3m13.690627221s) [noise=true , address=45.91.116.8:4046 , piece_id=UKLRKQMJJ3RDNCQ7E3TWZL3MPLFAHWA6CMLZPPTFJJNXIMGPQ5KQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.820027075 (568h3m13.691110131s) [noise=true , address=145.79.227.48:4301 , piece_id=3CYFNU64ELINNXLA2WUDNYEGGILOY6WYMU365KE3VXDAOE7VIN4Q, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.820479427 (568h3m13.691562462s) [noise=true , address=45.91.116.13:4153 , piece_id=IUMYOKZMJUVP2LUSY7S5TTNQBCF6LAOGKP2U6NBXMM2ZN3TWSLLQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.820801921 (568h3m13.691884966s) [noise=true , address=145.79.227.52:4445 , piece_id=RTCV457EQY6BQQGUBKDJSHXJPPGDNMGYAIFEQH4SRTPVHR7ENEXA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.822044126 (568h3m13.693127171s) [noise=true , address=45.91.116.14:4192 , piece_id=A5TU3EG7EKWTKABQVWQZQE4EX7UJ5MWB3XYQ3WEEOBU5TTUXPJSA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.822265227 (568h3m13.693348272s) [noise=true , address=45.91.116.9:4263 , piece_id=D7JFAXL7CDMK3Y2V6O6ZFO4B3FN4I4LO2CSKKDT7XC3XVSZZQKKQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.823803146 (568h3m13.694886181s) [noise=true , address=77.90.31.13:4626 , piece_id=XOSBBKNWP6VGWESKQFFT3572S4MNBPN5I4Z2R33C6S44LEPG4Y4Q, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.825561574 (568h3m13.696644619s) [noise=true , address=45.91.116.10:4083 , piece_id=DM7FQALVYY6B3OXSEOWORDJMYWFQQRI7PBGYXZHS3OINTYWFWKJA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.829174975 (568h3m13.70025801s) [noise=true , address=145.79.227.7:4295 , piece_id=Z7TC3PGCCD26G42HFKKIOBT7VXTLKOAWF44WGT7JBA2D2IZQIFHA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.833459964 (568h3m13.704543009s) [noise=true , address=45.91.116.7:4010 , piece_id=JI3LOKGGWOJOYFNIRFUFVYBHCBVG6BFJHVLQ7EF6F447OGU6SUZA, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.834279094 (568h3m13.705362149s) [noise=true , address=45.91.116.12:4225 , piece_id=DYTULOUH7OMZZWSHHLBAMMCE7AGYIWLLKFX22NNL4HONFAPPO6RQ, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:13.837406649 (568h3m13.708489684s) [noise=true , address=45.91.116.6:4974 , piece_id=4TN43QOOIJGNNBCI6ELIQYUKIYEWQ6YLIL572AXHJZHCXOJTT47A, seg_pos={0 0}, upload=498021]: Uploading piece... Dec 29 07:32:15.538987291 (568h3m15.410070336s) [upload=498021]: Next returned nil segment Dec 29 07:32:15.539002550 (568h3m15.410085595s) [upload=498021]: Waiting for error group managing segments... Dec 29 07:32:15.822958259 (568h3m15.694041305s) [noise=true , address=145.79.227.47:4266 , piece_id=C6BUKWRREDCSERVR4UPGSMKRXCTOQ6ZT24VQCSR54J5UKDOBUUJQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.823745880 (568h3m15.694828925s) [noise=true , address=145.79.227.42:4122 , piece_id=TJJS7WYFIVVVAJ2OO7GXO6VPDNU2N3EYLDW3UDA6ZEWKNBBMUQLA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.826646713 (568h3m15.697729748s) [noise=true , address=145.79.227.14:4838 , piece_id=62OG3EOAY4YMLFVSYHU6OF7Q46MJFVS5ZO6FOVVAO2GAKL45DWWQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.829780210 (568h3m15.700863245s) [noise=true , address=145.79.227.48:4301 , piece_id=3CYFNU64ELINNXLA2WUDNYEGGILOY6WYMU365KE3VXDAOE7VIN4Q, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.830236929 (568h3m15.701319964s) [noise=true , address=145.79.227.7:4295 , piece_id=Z7TC3PGCCD26G42HFKKIOBT7VXTLKOAWF44WGT7JBA2D2IZQIFHA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:15.839131236 (568h3m15.710214271s) [noise=true , address=145.79.227.52:4445 , piece_id=RTCV457EQY6BQQGUBKDJSHXJPPGDNMGYAIFEQH4SRTPVHR7ENEXA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.071461683 (568h3m15.942544728s) [noise=true , address=45.91.116.7:4010 , piece_id=JI3LOKGGWOJOYFNIRFUFVYBHCBVG6BFJHVLQ7EF6F447OGU6SUZA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.155399806 (568h3m16.026482851s) [noise=true , address=77.90.31.13:4626 , piece_id=XOSBBKNWP6VGWESKQFFT3572S4MNBPN5I4Z2R33C6S44LEPG4Y4Q, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.184915247 (568h3m16.055998292s) [noise=true , address=45.91.116.14:4192 , piece_id=A5TU3EG7EKWTKABQVWQZQE4EX7UJ5MWB3XYQ3WEEOBU5TTUXPJSA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.190219708 (568h3m16.061302754s) [noise=true , address=45.91.116.8:4046 , piece_id=UKLRKQMJJ3RDNCQ7E3TWZL3MPLFAHWA6CMLZPPTFJJNXIMGPQ5KQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.220678457 (568h3m16.091761492s) [noise=true , address=45.91.116.6:4974 , piece_id=4TN43QOOIJGNNBCI6ELIQYUKIYEWQ6YLIL572AXHJZHCXOJTT47A, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.273564812 (568h3m16.144647847s) [noise=true , address=45.91.116.10:4083 , piece_id=DM7FQALVYY6B3OXSEOWORDJMYWFQQRI7PBGYXZHS3OINTYWFWKJA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.757690240 (568h3m16.628773285s) [noise=true , address=77.90.31.19:4841 , piece_id=X3YPQ6QLWSOHCPAZQDEA3OGCYDGIIOIUAYYWJOW62637T2WOG7KA, seg_pos={0 0}, upload=498021]: Done uploading piece. err: <nil> Dec 29 07:32:16.757720176 (568h3m16.628803211s) [seg_pos={0 0}, upload=498021]: Segment reached optimal threshold of 13 pieces. Dec 29 07:32:16.758190502 (568h3m16.629273547s) [noise=true , address=45.91.116.12:4225 , piece_id=DYTULOUH7OMZZWSHHLBAMMCE7AGYIWLLKFX22NNL4HONFAPPO6RQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 07:32:16.758206211 (568h3m16.629289246s) [noise=true , address=45.91.116.12:4225 , piece_id=DYTULOUH7OMZZWSHHLBAMMCE7AGYIWLLKFX22NNL4HONFAPPO6RQ, seg_pos={0 0}, upload=498021]: Long tail cancellation due to optimal threshold Dec 29 07:32:16.758288419 (568h3m16.629371454s) [noise=true , address=45.91.116.13:4153 , piece_id=IUMYOKZMJUVP2LUSY7S5TTNQBCF6LAOGKP2U6NBXMM2ZN3TWSLLQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:45698->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:45698->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45698->45.91.116.13:4153: use of closed network connection Dec 29 07:32:16.758298758 (568h3m16.629381803s) [noise=true , address=45.91.116.13:4153 , piece_id=IUMYOKZMJUVP2LUSY7S5TTNQBCF6LAOGKP2U6NBXMM2ZN3TWSLLQ, seg_pos={0 0}, upload=498021]: Long tail cancellation due to optimal threshold Dec 29 07:32:16.758441339 (568h3m16.629524374s) [noise=true , address=45.91.116.9:4263 , piece_id=D7JFAXL7CDMK3Y2V6O6ZFO4B3FN4I4LO2CSKKDT7XC3XVSZZQKKQ, seg_pos={0 0}, upload=498021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 29 07:32:16.758450486 (568h3m16.629533522s) [noise=true , address=45.91.116.9:4263 , piece_id=D7JFAXL7CDMK3Y2V6O6ZFO4B3FN4I4LO2CSKKDT7XC3XVSZZQKKQ, seg_pos={0 0}, upload=498021]: Long tail cancellation due to optimal threshold Dec 29 07:32:16.758473270 (568h3m16.629556305s) [seg_pos={0 0}, upload=498021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:32:16.758640359 (568h3m16.629723394s) [upload=498021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:32:16.758647021 (568h3m16.629730066s) [upload=498021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:32:17.054367334 (568h3m16.925450379s) [upload=498021]: Done uploading segments... Dec 29 07:32:17.054392401 (568h3m16.925475456s) [upload=498021]: Upload finished. err: <nil> Duration: 30.003806887s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X.......X...........................................] duration: 4.882396507s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:32:13 UTC) orbiter warning: context deadline exceeded Duration: 4.882428648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X......X...........................X......] duration: 31.210660872s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:32:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 31.210688947s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X.......X..........................................] duration: 4.188838664s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:31:54 UTC) orbiter warning: context deadline exceeded Duration: 4.188862259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXX.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X...X..X] duration: 30.002354063s trace: https://jaeger.dev.storj.io/trace/326c066ad81cbe4c FAIL (2025-12-29 07:31:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 07:31:02.723237076 (568h2m2.594320121s) [upload=498004]: Starting upload Dec 29 07:31:02.723425445 (568h2m2.59450848s) [upload=498004]: Done starting upload Dec 29 07:31:02.723480470 (568h2m2.594563515s) [upload=498004]: Uploading segments... Dec 29 07:31:02.723674950 (568h2m2.594757985s) [upload=498004]: Got next segment. Inline: false Dec 29 07:31:02.723726920 (568h2m2.594809965s) [upload=498004]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:31:02.723733202 (568h2m2.594816247s) [upload=498004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:31:02.970608544 (568h2m2.841691589s) [seg_pos={0 0}, upload=498004]: Begin upload segment... Dec 29 07:31:02.970676984 (568h2m2.841760029s) [seg_pos={0 0}, upload=498004]: Done begin upload segment. Dec 29 07:31:02.970720517 (568h2m2.841803562s) [noise=true , address=145.79.227.44:4195 , piece_id=RXSBJTIWJJJMJGZS22DKQOVXUGIOUSSXU2JCWVPYRKEYX6ZWJO4Q, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.970733713 (568h2m2.841816758s) [noise=true , address=45.91.116.8:4045 , piece_id=P664AFAAVZGDMQ6JYWQBPGDLS5EEXWC5BFNSMIVXS55DSNR3J2KQ, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.971828888 (568h2m2.842911933s) [noise=true , address=145.79.227.12:4765 , piece_id=UJXFKKFMXVWT7NB4ONFKBON5RUMU4R6X5SBBOVDTBDAHOEVHISLA, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.972312068 (568h2m2.843395113s) [noise=true , address=45.91.116.9:4263 , piece_id=C4EECON4BXHOSSHY23PUG3AU7OE27EDLGZN7LTS64GVIQ5BHWX5Q, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.972806599 (568h2m2.843889654s) [noise=true , address=145.79.227.43:4159 , piece_id=TABIGTWKAPNH2T7ABPKIBXXWEJRIMGW2PGJ465FZVNX3YLGYCI3A, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.973255063 (568h2m2.844338128s) [noise=true , address=145.79.227.35:4483 , piece_id=MYNLHYHL7RA4DOX5QHOFFFTT2HEMAGRWL52C6RYRQWXE2XBQKFZA, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.973730539 (568h2m2.844813584s) [noise=true , address=145.79.227.18:4007 , piece_id=OZX3ELDFGQFFI7ACXN27WKA4JN3PI22AZBA5YJRDJWWBTXPJ3NRQ, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.974070086 (568h2m2.845153121s) [noise=true , address=145.79.227.39:4014 , piece_id=XH4ZAXJVPR67RFLFLGW6GK72J7O4PMQEB355RDFS5HU4IOWRCBNA, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.974792190 (568h2m2.845875225s) [noise=true , address=45.91.116.11:4117 , piece_id=PIHBCTQCKRCDE7TGLAJ2P7RFI33KBZ4C6GXSPHV3YCQYREBXY5MA, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.975339432 (568h2m2.846422478s) [noise=true , address=145.79.227.13:4804 , piece_id=CMKLG6C4HVNFXQGEWQGQUMZYEOG64YUBYTS75I22QZ3NZRKDASCQ, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.975891394 (568h2m2.846974449s) [noise=true , address=145.79.227.26:4654 , piece_id=4U57CY5QOQSYL3JEX7OIDLV2HV7ZCOH6QXNPAFXQYSYRY5NH7F6A, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.976619881 (568h2m2.847702926s) [noise=true , address=145.79.227.51:4409 , piece_id=WDHOAO3TLGZ3AAU76YX56C564SCBS6NL2MHOLRYW4YODHR6ZQBNA, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.977181480 (568h2m2.848264525s) [noise=true , address=145.79.227.4:4187 , piece_id=FXFR6AEWJYGUOEG2YUDVTGESUFGGPKVB4UVEKFRBXLFWIL64GMRA, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.978584261 (568h2m2.849667306s) [noise=true , address=145.79.227.36:4518 , piece_id=DJIBQASK4U5H54MXOND34O3BDAHMITAKU4NPFYA3W3A5CB6XVPAA, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.980444624 (568h2m2.851527679s) [noise=true , address=145.79.227.38:4977 , piece_id=IOBRELFHJZWE4EHREK6WBDLHKUSCN2YVOZQYRKW2TFSFA7H6YKDQ, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:02.982916361 (568h2m2.853999426s) [noise=true , address=145.79.227.22:4509 , piece_id=JFVGHSKDXE5S7QO2Y4AYOIV7G4M5P5UFUHFVC5N3BEFRACQG4KFQ, seg_pos={0 0}, upload=498004]: Uploading piece... Dec 29 07:31:04.682238579 (568h2m4.553321614s) [upload=498004]: Next returned nil segment Dec 29 07:31:04.682292302 (568h2m4.553375347s) [upload=498004]: Waiting for error group managing segments... Dec 29 07:31:04.971721966 (568h2m4.842805001s) [noise=true , address=145.79.227.12:4765 , piece_id=UJXFKKFMXVWT7NB4ONFKBON5RUMU4R6X5SBBOVDTBDAHOEVHISLA, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:04.974103240 (568h2m4.845186275s) [noise=true , address=145.79.227.38:4977 , piece_id=IOBRELFHJZWE4EHREK6WBDLHKUSCN2YVOZQYRKW2TFSFA7H6YKDQ, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:04.975651829 (568h2m4.846734864s) [noise=true , address=145.79.227.18:4007 , piece_id=OZX3ELDFGQFFI7ACXN27WKA4JN3PI22AZBA5YJRDJWWBTXPJ3NRQ, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:04.980244865 (568h2m4.85132791s) [noise=true , address=145.79.227.51:4409 , piece_id=WDHOAO3TLGZ3AAU76YX56C564SCBS6NL2MHOLRYW4YODHR6ZQBNA, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:04.982819087 (568h2m4.853902132s) [noise=true , address=145.79.227.13:4804 , piece_id=CMKLG6C4HVNFXQGEWQGQUMZYEOG64YUBYTS75I22QZ3NZRKDASCQ, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:04.985786247 (568h2m4.856869292s) [noise=true , address=145.79.227.39:4014 , piece_id=XH4ZAXJVPR67RFLFLGW6GK72J7O4PMQEB355RDFS5HU4IOWRCBNA, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:04.989095217 (568h2m4.860178263s) [noise=true , address=145.79.227.26:4654 , piece_id=4U57CY5QOQSYL3JEX7OIDLV2HV7ZCOH6QXNPAFXQYSYRY5NH7F6A, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:04.993606437 (568h2m4.864689482s) [noise=true , address=145.79.227.22:4509 , piece_id=JFVGHSKDXE5S7QO2Y4AYOIV7G4M5P5UFUHFVC5N3BEFRACQG4KFQ, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:04.994162316 (568h2m4.865245361s) [noise=true , address=145.79.227.36:4518 , piece_id=DJIBQASK4U5H54MXOND34O3BDAHMITAKU4NPFYA3W3A5CB6XVPAA, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:04.995301114 (568h2m4.866384149s) [noise=true , address=145.79.227.4:4187 , piece_id=FXFR6AEWJYGUOEG2YUDVTGESUFGGPKVB4UVEKFRBXLFWIL64GMRA, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:05.222036086 (568h2m5.093119132s) [noise=true , address=45.91.116.9:4263 , piece_id=C4EECON4BXHOSSHY23PUG3AU7OE27EDLGZN7LTS64GVIQ5BHWX5Q, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:05.244353332 (568h2m5.115436377s) [noise=true , address=45.91.116.11:4117 , piece_id=PIHBCTQCKRCDE7TGLAJ2P7RFI33KBZ4C6GXSPHV3YCQYREBXY5MA, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:06.239136990 (568h2m6.110220025s) [noise=true , address=145.79.227.44:4195 , piece_id=RXSBJTIWJJJMJGZS22DKQOVXUGIOUSSXU2JCWVPYRKEYX6ZWJO4Q, seg_pos={0 0}, upload=498004]: Done uploading piece. err: <nil> Dec 29 07:31:06.239164062 (568h2m6.110247107s) [seg_pos={0 0}, upload=498004]: Segment reached optimal threshold of 13 pieces. Dec 29 07:31:06.239459815 (568h2m6.11054286s) [noise=true , address=145.79.227.43:4159 , piece_id=TABIGTWKAPNH2T7ABPKIBXXWEJRIMGW2PGJ465FZVNX3YLGYCI3A, seg_pos={0 0}, upload=498004]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 07:31:06.239471547 (568h2m6.110554592s) [noise=true , address=145.79.227.43:4159 , piece_id=TABIGTWKAPNH2T7ABPKIBXXWEJRIMGW2PGJ465FZVNX3YLGYCI3A, seg_pos={0 0}, upload=498004]: Long tail cancellation due to optimal threshold Dec 29 07:31:06.239656099 (568h2m6.110739154s) [noise=true , address=145.79.227.35:4483 , piece_id=MYNLHYHL7RA4DOX5QHOFFFTT2HEMAGRWL52C6RYRQWXE2XBQKFZA, seg_pos={0 0}, upload=498004]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 07:31:06.239664815 (568h2m6.11074786s) [noise=true , address=145.79.227.35:4483 , piece_id=MYNLHYHL7RA4DOX5QHOFFFTT2HEMAGRWL52C6RYRQWXE2XBQKFZA, seg_pos={0 0}, upload=498004]: Long tail cancellation due to optimal threshold Dec 29 07:31:06.239809230 (568h2m6.110892265s) [noise=true , address=45.91.116.8:4045 , piece_id=P664AFAAVZGDMQ6JYWQBPGDLS5EEXWC5BFNSMIVXS55DSNR3J2KQ, seg_pos={0 0}, upload=498004]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 07:31:06.239828487 (568h2m6.110911532s) [noise=true , address=45.91.116.8:4045 , piece_id=P664AFAAVZGDMQ6JYWQBPGDLS5EEXWC5BFNSMIVXS55DSNR3J2KQ, seg_pos={0 0}, upload=498004]: Long tail cancellation due to optimal threshold Dec 29 07:31:06.239868413 (568h2m6.110951448s) [seg_pos={0 0}, upload=498004]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:31:06.239994743 (568h2m6.111077788s) [upload=498004]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:31:06.240004352 (568h2m6.111087387s) [upload=498004]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:31:06.530768747 (568h2m6.401851782s) [upload=498004]: Done uploading segments... Dec 29 07:31:06.530790208 (568h2m6.401873243s) [upload=498004]: Upload finished. err: <nil> Duration: 30.002381586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X......X...........................X.....] duration: 35.708489527s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:31:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 35.708517963s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X.......X.........................................] duration: 3.475498063s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:31:09 UTC) orbiter warning: context deadline exceeded Duration: 3.475528329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X......X...........................X....] duration: 15.42244635s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:30:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 15.422479373s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X.......X........................................] duration: 3.513809603s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:30:26 UTC) orbiter warning: context deadline exceeded Duration: 3.513841303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXXXX.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X...X..] duration: 6.263462763s trace: https://jaeger.dev.storj.io/trace/23d8b13b5fba40d1 FAIL (2025-12-29 07:30:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: YI3PAPVNFAOFCTQYX2RDKJTQBRNEBT2NADUZ7Z25SRMXACZBJ6UQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SFYW3VRP6QR2BZPGDBQSQ55U4RUHMSFECFU55OCOUDWVNSSRCKGQ) context deadline exceeded ======LOGS====== (567h59m34.935482206s) [upload=497969]: Done starting upload Dec 29 07:28:35.064506646 (567h59m34.935589691s) [upload=497969]: Uploading segments... Dec 29 07:28:35.064613700 (567h59m34.935696735s) [upload=497969]: Got next segment. Inline: false Dec 29 07:28:35.064665758 (567h59m34.935748803s) [upload=497969]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:28:35.064672220 (567h59m34.935755265s) [upload=497969]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:28:35.309553878 (567h59m35.180636913s) [seg_pos={0 0}, upload=497969]: Begin upload segment... Dec 29 07:28:35.309600066 (567h59m35.180683101s) [seg_pos={0 0}, upload=497969]: Done begin upload segment. Dec 29 07:28:35.309623751 (567h59m35.180706786s) [noise=true , address=145.79.227.39:4014 , piece_id=RT4DZ4ODKQFLYYVIJN7JMDNBZ7BVS2HBYZJYGN627FBRZQHSWVUA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.309728430 (567h59m35.180811465s) [noise=true , address=145.79.227.45:4231 , piece_id=JBZMYACIS65APR6DF6R4CEXFQHPUO3JSNH62TWOIPLSLASPBSU5Q, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.310192534 (567h59m35.181275569s) [noise=true , address=77.90.31.21:4880 , piece_id=SFYW3VRP6QR2BZPGDBQSQ55U4RUHMSFECFU55OCOUDWVNSSRCKGQ, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.310502004 (567h59m35.181585059s) [noise=true , address=45.91.116.9:4261 , piece_id=LBDMMQ2AIIQVL3CBKHQSWQLSHK24GT2E4TGVTGL4PNYBIOAXQMPQ, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.310943724 (567h59m35.182026759s) [noise=true , address=145.79.227.14:4840 , piece_id=7UJIKBMVVMFKIVESVOCBUDMTZ43QAINA3IL2QUMTLYTEPPWOXY2Q, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.311351842 (567h59m35.182434887s) [noise=true , address=145.79.227.26:4654 , piece_id=L63SDM7IIUF375GKRKFDIINWGDFMCHK5TI3NFFRIS53P46MUJXCA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.311693663 (567h59m35.182776708s) [noise=true , address=145.79.227.36:4517 , piece_id=52M42UDW4P56MJGZRDK5BREACWDMNSHFIOES5UBXWM5DNHIITZ2A, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.313104358 (567h59m35.184187403s) [noise=true , address=145.79.227.4:4187 , piece_id=2XXYHEDBEH7IIATGDHQEGKEQPMTE2DVOXC6K4XH5LWLSECRUHA3Q, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.314378034 (567h59m35.185461069s) [noise=true , address=145.79.227.47:4265 , piece_id=GY5VYCH2YLREPCLGIUTU45MPMSB6ATS4ZOUNOSFJB3OFMTWHAGOA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.315463811 (567h59m35.186546846s) [noise=true , address=77.90.31.11:4556 , piece_id=7VZCECY6U4FT4J3WMZMY4UKAAIVKP2F2ESAJLV4URD74H25FR5MA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.316512769 (567h59m35.187595814s) [noise=true , address=145.79.227.12:4767 , piece_id=ZWQ4WMXX3WH55NVMT2YZWIJM2KK27NFCDQGCN6LW6GYDHLFI4NGQ, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.317591673 (567h59m35.188674708s) [noise=true , address=77.90.31.14:4663 , piece_id=YI3PAPVNFAOFCTQYX2RDKJTQBRNEBT2NADUZ7Z25SRMXACZBJ6UQ, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.318567812 (567h59m35.189650847s) [noise=true , address=145.79.227.48:4301 , piece_id=YQYLTAKUPFVPMDB6ZC5C3LY4T243UBZLWSHIC3B43RHQWHHDCR4A, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.319683336 (567h59m35.190766371s) [noise=true , address=145.79.227.17:4970 , piece_id=6QSOH3ZI7HL5DAX5AHS7IXOJXMUDCEJM2UNZ2AQ6F7WNRZFJN64Q, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.320426872 (567h59m35.191509917s) [noise=true , address=145.79.227.13:4802 , piece_id=I3FTQ2TBD42HEBMNFQQAIFTZC2OYM5L3PMYZUCPJYCUOUHBNNYJA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.321633209 (567h59m35.192716244s) [noise=true , address=45.91.116.5:4297 , piece_id=VHCPQY55G4PS7NGZHO2W6AG4WXAVGAHU6PPTH5WS4ZU6GAUVGGXA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:36.914603170 (567h59m36.785686205s) [upload=497969]: Next returned nil segment Dec 29 07:28:36.914617186 (567h59m36.785700221s) [upload=497969]: Waiting for error group managing segments... Dec 29 07:28:37.291981242 (567h59m37.163064297s) [noise=true , address=145.79.227.48:4301 , piece_id=YQYLTAKUPFVPMDB6ZC5C3LY4T243UBZLWSHIC3B43RHQWHHDCR4A, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.297271045 (567h59m37.16835409s) [noise=true , address=145.79.227.14:4840 , piece_id=7UJIKBMVVMFKIVESVOCBUDMTZ43QAINA3IL2QUMTLYTEPPWOXY2Q, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.297981978 (567h59m37.169065013s) [noise=true , address=145.79.227.36:4517 , piece_id=52M42UDW4P56MJGZRDK5BREACWDMNSHFIOES5UBXWM5DNHIITZ2A, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.298675228 (567h59m37.169758273s) [noise=true , address=145.79.227.39:4014 , piece_id=RT4DZ4ODKQFLYYVIJN7JMDNBZ7BVS2HBYZJYGN627FBRZQHSWVUA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.299022710 (567h59m37.170105755s) [noise=true , address=145.79.227.45:4231 , piece_id=JBZMYACIS65APR6DF6R4CEXFQHPUO3JSNH62TWOIPLSLASPBSU5Q, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.300251961 (567h59m37.171334996s) [noise=true , address=145.79.227.12:4767 , piece_id=ZWQ4WMXX3WH55NVMT2YZWIJM2KK27NFCDQGCN6LW6GYDHLFI4NGQ, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.306268898 (567h59m37.177351933s) [noise=true , address=145.79.227.17:4970 , piece_id=6QSOH3ZI7HL5DAX5AHS7IXOJXMUDCEJM2UNZ2AQ6F7WNRZFJN64Q, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.309168289 (567h59m37.180251334s) [noise=true , address=145.79.227.4:4187 , piece_id=2XXYHEDBEH7IIATGDHQEGKEQPMTE2DVOXC6K4XH5LWLSECRUHA3Q, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.314813208 (567h59m37.185896253s) [noise=true , address=145.79.227.47:4265 , piece_id=GY5VYCH2YLREPCLGIUTU45MPMSB6ATS4ZOUNOSFJB3OFMTWHAGOA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.320837599 (567h59m37.191920644s) [noise=true , address=145.79.227.26:4654 , piece_id=L63SDM7IIUF375GKRKFDIINWGDFMCHK5TI3NFFRIS53P46MUJXCA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.324054144 (567h59m37.195137189s) [noise=true , address=145.79.227.13:4802 , piece_id=I3FTQ2TBD42HEBMNFQQAIFTZC2OYM5L3PMYZUCPJYCUOUHBNNYJA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.487402096 (567h59m37.358485131s) [noise=true , address=77.90.31.21:4880 , piece_id=SFYW3VRP6QR2BZPGDBQSQ55U4RUHMSFECFU55OCOUDWVNSSRCKGQ, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.489137731 (567h59m37.360220766s) [noise=true , address=77.90.31.14:4663 , piece_id=YI3PAPVNFAOFCTQYX2RDKJTQBRNEBT2NADUZ7Z25SRMXACZBJ6UQ, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.489154453 (567h59m37.360237499s) [seg_pos={0 0}, upload=497969]: Segment reached optimal threshold of 13 pieces. Dec 29 07:28:37.489426691 (567h59m37.360509736s) [noise=true , address=45.91.116.9:4261 , piece_id=LBDMMQ2AIIQVL3CBKHQSWQLSHK24GT2E4TGVTGL4PNYBIOAXQMPQ, seg_pos={0 0}, upload=497969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:54484->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:28:37.489435308 (567h59m37.360518343s) [noise=true , address=45.91.116.9:4261 , piece_id=LBDMMQ2AIIQVL3CBKHQSWQLSHK24GT2E4TGVTGL4PNYBIOAXQMPQ, seg_pos={0 0}, upload=497969]: Long tail cancellation due to optimal threshold Dec 29 07:28:37.489564013 (567h59m37.360647058s) [noise=true , address=45.91.116.5:4297 , piece_id=VHCPQY55G4PS7NGZHO2W6AG4WXAVGAHU6PPTH5WS4ZU6GAUVGGXA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:52868->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:52868->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52868->45.91.116.5:4297: use of closed network connection Dec 29 07:28:37.489572419 (567h59m37.360655464s) [noise=true , address=45.91.116.5:4297 , piece_id=VHCPQY55G4PS7NGZHO2W6AG4WXAVGAHU6PPTH5WS4ZU6GAUVGGXA, seg_pos={0 0}, upload=497969]: Long tail cancellation due to optimal threshold Dec 29 07:28:37.489609460 (567h59m37.360692495s) [noise=true , address=77.90.31.11:4556 , piece_id=7VZCECY6U4FT4J3WMZMY4UKAAIVKP2F2ESAJLV4URD74H25FR5MA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 07:28:37.489619539 (567h59m37.360702584s) [noise=true , address=77.90.31.11:4556 , piece_id=7VZCECY6U4FT4J3WMZMY4UKAAIVKP2F2ESAJLV4URD74H25FR5MA, seg_pos={0 0}, upload=497969]: Long tail cancellation due to optimal threshold Dec 29 07:28:37.489680545 (567h59m37.36076358s) [seg_pos={0 0}, upload=497969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:28:37.489835229 (567h59m37.360918264s) [upload=497969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:28:37.489842272 (567h59m37.360925307s) [upload=497969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:28:37.789935020 (567h59m37.661018055s) [upload=497969]: Done uploading segments... Dec 29 07:28:37.789953025 (567h59m37.66103606s) [upload=497969]: Upload finished. err: <nil> Duration: 6.263495254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X......X...........................X...] duration: 15.418506448s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:29:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 15.418532077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X......X...........................X..] duration: 33.230260707s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:29:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 33.230279621s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XXXXX.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X...X.] duration: 5.993694687s trace: https://jaeger.dev.storj.io/trace/23d8b13b5fba40d1 FAIL (2025-12-29 07:29:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: YI3PAPVNFAOFCTQYX2RDKJTQBRNEBT2NADUZ7Z25SRMXACZBJ6UQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SFYW3VRP6QR2BZPGDBQSQ55U4RUHMSFECFU55OCOUDWVNSSRCKGQ) context deadline exceeded ======LOGS====== (567h59m34.935482206s) [upload=497969]: Done starting upload Dec 29 07:28:35.064506646 (567h59m34.935589691s) [upload=497969]: Uploading segments... Dec 29 07:28:35.064613700 (567h59m34.935696735s) [upload=497969]: Got next segment. Inline: false Dec 29 07:28:35.064665758 (567h59m34.935748803s) [upload=497969]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:28:35.064672220 (567h59m34.935755265s) [upload=497969]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:28:35.309553878 (567h59m35.180636913s) [seg_pos={0 0}, upload=497969]: Begin upload segment... Dec 29 07:28:35.309600066 (567h59m35.180683101s) [seg_pos={0 0}, upload=497969]: Done begin upload segment. Dec 29 07:28:35.309623751 (567h59m35.180706786s) [noise=true , address=145.79.227.39:4014 , piece_id=RT4DZ4ODKQFLYYVIJN7JMDNBZ7BVS2HBYZJYGN627FBRZQHSWVUA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.309728430 (567h59m35.180811465s) [noise=true , address=145.79.227.45:4231 , piece_id=JBZMYACIS65APR6DF6R4CEXFQHPUO3JSNH62TWOIPLSLASPBSU5Q, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.310192534 (567h59m35.181275569s) [noise=true , address=77.90.31.21:4880 , piece_id=SFYW3VRP6QR2BZPGDBQSQ55U4RUHMSFECFU55OCOUDWVNSSRCKGQ, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.310502004 (567h59m35.181585059s) [noise=true , address=45.91.116.9:4261 , piece_id=LBDMMQ2AIIQVL3CBKHQSWQLSHK24GT2E4TGVTGL4PNYBIOAXQMPQ, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.310943724 (567h59m35.182026759s) [noise=true , address=145.79.227.14:4840 , piece_id=7UJIKBMVVMFKIVESVOCBUDMTZ43QAINA3IL2QUMTLYTEPPWOXY2Q, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.311351842 (567h59m35.182434887s) [noise=true , address=145.79.227.26:4654 , piece_id=L63SDM7IIUF375GKRKFDIINWGDFMCHK5TI3NFFRIS53P46MUJXCA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.311693663 (567h59m35.182776708s) [noise=true , address=145.79.227.36:4517 , piece_id=52M42UDW4P56MJGZRDK5BREACWDMNSHFIOES5UBXWM5DNHIITZ2A, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.313104358 (567h59m35.184187403s) [noise=true , address=145.79.227.4:4187 , piece_id=2XXYHEDBEH7IIATGDHQEGKEQPMTE2DVOXC6K4XH5LWLSECRUHA3Q, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.314378034 (567h59m35.185461069s) [noise=true , address=145.79.227.47:4265 , piece_id=GY5VYCH2YLREPCLGIUTU45MPMSB6ATS4ZOUNOSFJB3OFMTWHAGOA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.315463811 (567h59m35.186546846s) [noise=true , address=77.90.31.11:4556 , piece_id=7VZCECY6U4FT4J3WMZMY4UKAAIVKP2F2ESAJLV4URD74H25FR5MA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.316512769 (567h59m35.187595814s) [noise=true , address=145.79.227.12:4767 , piece_id=ZWQ4WMXX3WH55NVMT2YZWIJM2KK27NFCDQGCN6LW6GYDHLFI4NGQ, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.317591673 (567h59m35.188674708s) [noise=true , address=77.90.31.14:4663 , piece_id=YI3PAPVNFAOFCTQYX2RDKJTQBRNEBT2NADUZ7Z25SRMXACZBJ6UQ, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.318567812 (567h59m35.189650847s) [noise=true , address=145.79.227.48:4301 , piece_id=YQYLTAKUPFVPMDB6ZC5C3LY4T243UBZLWSHIC3B43RHQWHHDCR4A, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.319683336 (567h59m35.190766371s) [noise=true , address=145.79.227.17:4970 , piece_id=6QSOH3ZI7HL5DAX5AHS7IXOJXMUDCEJM2UNZ2AQ6F7WNRZFJN64Q, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.320426872 (567h59m35.191509917s) [noise=true , address=145.79.227.13:4802 , piece_id=I3FTQ2TBD42HEBMNFQQAIFTZC2OYM5L3PMYZUCPJYCUOUHBNNYJA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.321633209 (567h59m35.192716244s) [noise=true , address=45.91.116.5:4297 , piece_id=VHCPQY55G4PS7NGZHO2W6AG4WXAVGAHU6PPTH5WS4ZU6GAUVGGXA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:36.914603170 (567h59m36.785686205s) [upload=497969]: Next returned nil segment Dec 29 07:28:36.914617186 (567h59m36.785700221s) [upload=497969]: Waiting for error group managing segments... Dec 29 07:28:37.291981242 (567h59m37.163064297s) [noise=true , address=145.79.227.48:4301 , piece_id=YQYLTAKUPFVPMDB6ZC5C3LY4T243UBZLWSHIC3B43RHQWHHDCR4A, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.297271045 (567h59m37.16835409s) [noise=true , address=145.79.227.14:4840 , piece_id=7UJIKBMVVMFKIVESVOCBUDMTZ43QAINA3IL2QUMTLYTEPPWOXY2Q, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.297981978 (567h59m37.169065013s) [noise=true , address=145.79.227.36:4517 , piece_id=52M42UDW4P56MJGZRDK5BREACWDMNSHFIOES5UBXWM5DNHIITZ2A, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.298675228 (567h59m37.169758273s) [noise=true , address=145.79.227.39:4014 , piece_id=RT4DZ4ODKQFLYYVIJN7JMDNBZ7BVS2HBYZJYGN627FBRZQHSWVUA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.299022710 (567h59m37.170105755s) [noise=true , address=145.79.227.45:4231 , piece_id=JBZMYACIS65APR6DF6R4CEXFQHPUO3JSNH62TWOIPLSLASPBSU5Q, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.300251961 (567h59m37.171334996s) [noise=true , address=145.79.227.12:4767 , piece_id=ZWQ4WMXX3WH55NVMT2YZWIJM2KK27NFCDQGCN6LW6GYDHLFI4NGQ, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.306268898 (567h59m37.177351933s) [noise=true , address=145.79.227.17:4970 , piece_id=6QSOH3ZI7HL5DAX5AHS7IXOJXMUDCEJM2UNZ2AQ6F7WNRZFJN64Q, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.309168289 (567h59m37.180251334s) [noise=true , address=145.79.227.4:4187 , piece_id=2XXYHEDBEH7IIATGDHQEGKEQPMTE2DVOXC6K4XH5LWLSECRUHA3Q, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.314813208 (567h59m37.185896253s) [noise=true , address=145.79.227.47:4265 , piece_id=GY5VYCH2YLREPCLGIUTU45MPMSB6ATS4ZOUNOSFJB3OFMTWHAGOA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.320837599 (567h59m37.191920644s) [noise=true , address=145.79.227.26:4654 , piece_id=L63SDM7IIUF375GKRKFDIINWGDFMCHK5TI3NFFRIS53P46MUJXCA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.324054144 (567h59m37.195137189s) [noise=true , address=145.79.227.13:4802 , piece_id=I3FTQ2TBD42HEBMNFQQAIFTZC2OYM5L3PMYZUCPJYCUOUHBNNYJA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.487402096 (567h59m37.358485131s) [noise=true , address=77.90.31.21:4880 , piece_id=SFYW3VRP6QR2BZPGDBQSQ55U4RUHMSFECFU55OCOUDWVNSSRCKGQ, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.489137731 (567h59m37.360220766s) [noise=true , address=77.90.31.14:4663 , piece_id=YI3PAPVNFAOFCTQYX2RDKJTQBRNEBT2NADUZ7Z25SRMXACZBJ6UQ, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.489154453 (567h59m37.360237499s) [seg_pos={0 0}, upload=497969]: Segment reached optimal threshold of 13 pieces. Dec 29 07:28:37.489426691 (567h59m37.360509736s) [noise=true , address=45.91.116.9:4261 , piece_id=LBDMMQ2AIIQVL3CBKHQSWQLSHK24GT2E4TGVTGL4PNYBIOAXQMPQ, seg_pos={0 0}, upload=497969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:54484->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:28:37.489435308 (567h59m37.360518343s) [noise=true , address=45.91.116.9:4261 , piece_id=LBDMMQ2AIIQVL3CBKHQSWQLSHK24GT2E4TGVTGL4PNYBIOAXQMPQ, seg_pos={0 0}, upload=497969]: Long tail cancellation due to optimal threshold Dec 29 07:28:37.489564013 (567h59m37.360647058s) [noise=true , address=45.91.116.5:4297 , piece_id=VHCPQY55G4PS7NGZHO2W6AG4WXAVGAHU6PPTH5WS4ZU6GAUVGGXA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:52868->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:52868->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52868->45.91.116.5:4297: use of closed network connection Dec 29 07:28:37.489572419 (567h59m37.360655464s) [noise=true , address=45.91.116.5:4297 , piece_id=VHCPQY55G4PS7NGZHO2W6AG4WXAVGAHU6PPTH5WS4ZU6GAUVGGXA, seg_pos={0 0}, upload=497969]: Long tail cancellation due to optimal threshold Dec 29 07:28:37.489609460 (567h59m37.360692495s) [noise=true , address=77.90.31.11:4556 , piece_id=7VZCECY6U4FT4J3WMZMY4UKAAIVKP2F2ESAJLV4URD74H25FR5MA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 07:28:37.489619539 (567h59m37.360702584s) [noise=true , address=77.90.31.11:4556 , piece_id=7VZCECY6U4FT4J3WMZMY4UKAAIVKP2F2ESAJLV4URD74H25FR5MA, seg_pos={0 0}, upload=497969]: Long tail cancellation due to optimal threshold Dec 29 07:28:37.489680545 (567h59m37.36076358s) [seg_pos={0 0}, upload=497969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:28:37.489835229 (567h59m37.360918264s) [upload=497969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:28:37.489842272 (567h59m37.360925307s) [upload=497969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:28:37.789935020 (567h59m37.661018055s) [upload=497969]: Done uploading segments... Dec 29 07:28:37.789953025 (567h59m37.66103606s) [upload=497969]: Upload finished. err: <nil> Duration: 5.993730545s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X.......X.......................................] duration: 3.495550496s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:29:13 UTC) orbiter warning: context deadline exceeded Duration: 3.495721481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.XXXXX.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X...X] duration: 30.002973462s trace: https://jaeger.dev.storj.io/trace/23d8b13b5fba40d1 FAIL (2025-12-29 07:29:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: YI3PAPVNFAOFCTQYX2RDKJTQBRNEBT2NADUZ7Z25SRMXACZBJ6UQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SFYW3VRP6QR2BZPGDBQSQ55U4RUHMSFECFU55OCOUDWVNSSRCKGQ) context deadline exceeded ======LOGS====== (567h59m34.935482206s) [upload=497969]: Done starting upload Dec 29 07:28:35.064506646 (567h59m34.935589691s) [upload=497969]: Uploading segments... Dec 29 07:28:35.064613700 (567h59m34.935696735s) [upload=497969]: Got next segment. Inline: false Dec 29 07:28:35.064665758 (567h59m34.935748803s) [upload=497969]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:28:35.064672220 (567h59m34.935755265s) [upload=497969]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:28:35.309553878 (567h59m35.180636913s) [seg_pos={0 0}, upload=497969]: Begin upload segment... Dec 29 07:28:35.309600066 (567h59m35.180683101s) [seg_pos={0 0}, upload=497969]: Done begin upload segment. Dec 29 07:28:35.309623751 (567h59m35.180706786s) [noise=true , address=145.79.227.39:4014 , piece_id=RT4DZ4ODKQFLYYVIJN7JMDNBZ7BVS2HBYZJYGN627FBRZQHSWVUA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.309728430 (567h59m35.180811465s) [noise=true , address=145.79.227.45:4231 , piece_id=JBZMYACIS65APR6DF6R4CEXFQHPUO3JSNH62TWOIPLSLASPBSU5Q, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.310192534 (567h59m35.181275569s) [noise=true , address=77.90.31.21:4880 , piece_id=SFYW3VRP6QR2BZPGDBQSQ55U4RUHMSFECFU55OCOUDWVNSSRCKGQ, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.310502004 (567h59m35.181585059s) [noise=true , address=45.91.116.9:4261 , piece_id=LBDMMQ2AIIQVL3CBKHQSWQLSHK24GT2E4TGVTGL4PNYBIOAXQMPQ, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.310943724 (567h59m35.182026759s) [noise=true , address=145.79.227.14:4840 , piece_id=7UJIKBMVVMFKIVESVOCBUDMTZ43QAINA3IL2QUMTLYTEPPWOXY2Q, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.311351842 (567h59m35.182434887s) [noise=true , address=145.79.227.26:4654 , piece_id=L63SDM7IIUF375GKRKFDIINWGDFMCHK5TI3NFFRIS53P46MUJXCA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.311693663 (567h59m35.182776708s) [noise=true , address=145.79.227.36:4517 , piece_id=52M42UDW4P56MJGZRDK5BREACWDMNSHFIOES5UBXWM5DNHIITZ2A, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.313104358 (567h59m35.184187403s) [noise=true , address=145.79.227.4:4187 , piece_id=2XXYHEDBEH7IIATGDHQEGKEQPMTE2DVOXC6K4XH5LWLSECRUHA3Q, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.314378034 (567h59m35.185461069s) [noise=true , address=145.79.227.47:4265 , piece_id=GY5VYCH2YLREPCLGIUTU45MPMSB6ATS4ZOUNOSFJB3OFMTWHAGOA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.315463811 (567h59m35.186546846s) [noise=true , address=77.90.31.11:4556 , piece_id=7VZCECY6U4FT4J3WMZMY4UKAAIVKP2F2ESAJLV4URD74H25FR5MA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.316512769 (567h59m35.187595814s) [noise=true , address=145.79.227.12:4767 , piece_id=ZWQ4WMXX3WH55NVMT2YZWIJM2KK27NFCDQGCN6LW6GYDHLFI4NGQ, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.317591673 (567h59m35.188674708s) [noise=true , address=77.90.31.14:4663 , piece_id=YI3PAPVNFAOFCTQYX2RDKJTQBRNEBT2NADUZ7Z25SRMXACZBJ6UQ, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.318567812 (567h59m35.189650847s) [noise=true , address=145.79.227.48:4301 , piece_id=YQYLTAKUPFVPMDB6ZC5C3LY4T243UBZLWSHIC3B43RHQWHHDCR4A, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.319683336 (567h59m35.190766371s) [noise=true , address=145.79.227.17:4970 , piece_id=6QSOH3ZI7HL5DAX5AHS7IXOJXMUDCEJM2UNZ2AQ6F7WNRZFJN64Q, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.320426872 (567h59m35.191509917s) [noise=true , address=145.79.227.13:4802 , piece_id=I3FTQ2TBD42HEBMNFQQAIFTZC2OYM5L3PMYZUCPJYCUOUHBNNYJA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:35.321633209 (567h59m35.192716244s) [noise=true , address=45.91.116.5:4297 , piece_id=VHCPQY55G4PS7NGZHO2W6AG4WXAVGAHU6PPTH5WS4ZU6GAUVGGXA, seg_pos={0 0}, upload=497969]: Uploading piece... Dec 29 07:28:36.914603170 (567h59m36.785686205s) [upload=497969]: Next returned nil segment Dec 29 07:28:36.914617186 (567h59m36.785700221s) [upload=497969]: Waiting for error group managing segments... Dec 29 07:28:37.291981242 (567h59m37.163064297s) [noise=true , address=145.79.227.48:4301 , piece_id=YQYLTAKUPFVPMDB6ZC5C3LY4T243UBZLWSHIC3B43RHQWHHDCR4A, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.297271045 (567h59m37.16835409s) [noise=true , address=145.79.227.14:4840 , piece_id=7UJIKBMVVMFKIVESVOCBUDMTZ43QAINA3IL2QUMTLYTEPPWOXY2Q, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.297981978 (567h59m37.169065013s) [noise=true , address=145.79.227.36:4517 , piece_id=52M42UDW4P56MJGZRDK5BREACWDMNSHFIOES5UBXWM5DNHIITZ2A, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.298675228 (567h59m37.169758273s) [noise=true , address=145.79.227.39:4014 , piece_id=RT4DZ4ODKQFLYYVIJN7JMDNBZ7BVS2HBYZJYGN627FBRZQHSWVUA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.299022710 (567h59m37.170105755s) [noise=true , address=145.79.227.45:4231 , piece_id=JBZMYACIS65APR6DF6R4CEXFQHPUO3JSNH62TWOIPLSLASPBSU5Q, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.300251961 (567h59m37.171334996s) [noise=true , address=145.79.227.12:4767 , piece_id=ZWQ4WMXX3WH55NVMT2YZWIJM2KK27NFCDQGCN6LW6GYDHLFI4NGQ, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.306268898 (567h59m37.177351933s) [noise=true , address=145.79.227.17:4970 , piece_id=6QSOH3ZI7HL5DAX5AHS7IXOJXMUDCEJM2UNZ2AQ6F7WNRZFJN64Q, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.309168289 (567h59m37.180251334s) [noise=true , address=145.79.227.4:4187 , piece_id=2XXYHEDBEH7IIATGDHQEGKEQPMTE2DVOXC6K4XH5LWLSECRUHA3Q, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.314813208 (567h59m37.185896253s) [noise=true , address=145.79.227.47:4265 , piece_id=GY5VYCH2YLREPCLGIUTU45MPMSB6ATS4ZOUNOSFJB3OFMTWHAGOA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.320837599 (567h59m37.191920644s) [noise=true , address=145.79.227.26:4654 , piece_id=L63SDM7IIUF375GKRKFDIINWGDFMCHK5TI3NFFRIS53P46MUJXCA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.324054144 (567h59m37.195137189s) [noise=true , address=145.79.227.13:4802 , piece_id=I3FTQ2TBD42HEBMNFQQAIFTZC2OYM5L3PMYZUCPJYCUOUHBNNYJA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.487402096 (567h59m37.358485131s) [noise=true , address=77.90.31.21:4880 , piece_id=SFYW3VRP6QR2BZPGDBQSQ55U4RUHMSFECFU55OCOUDWVNSSRCKGQ, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.489137731 (567h59m37.360220766s) [noise=true , address=77.90.31.14:4663 , piece_id=YI3PAPVNFAOFCTQYX2RDKJTQBRNEBT2NADUZ7Z25SRMXACZBJ6UQ, seg_pos={0 0}, upload=497969]: Done uploading piece. err: <nil> Dec 29 07:28:37.489154453 (567h59m37.360237499s) [seg_pos={0 0}, upload=497969]: Segment reached optimal threshold of 13 pieces. Dec 29 07:28:37.489426691 (567h59m37.360509736s) [noise=true , address=45.91.116.9:4261 , piece_id=LBDMMQ2AIIQVL3CBKHQSWQLSHK24GT2E4TGVTGL4PNYBIOAXQMPQ, seg_pos={0 0}, upload=497969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:54484->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:28:37.489435308 (567h59m37.360518343s) [noise=true , address=45.91.116.9:4261 , piece_id=LBDMMQ2AIIQVL3CBKHQSWQLSHK24GT2E4TGVTGL4PNYBIOAXQMPQ, seg_pos={0 0}, upload=497969]: Long tail cancellation due to optimal threshold Dec 29 07:28:37.489564013 (567h59m37.360647058s) [noise=true , address=45.91.116.5:4297 , piece_id=VHCPQY55G4PS7NGZHO2W6AG4WXAVGAHU6PPTH5WS4ZU6GAUVGGXA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:52868->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:52868->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52868->45.91.116.5:4297: use of closed network connection Dec 29 07:28:37.489572419 (567h59m37.360655464s) [noise=true , address=45.91.116.5:4297 , piece_id=VHCPQY55G4PS7NGZHO2W6AG4WXAVGAHU6PPTH5WS4ZU6GAUVGGXA, seg_pos={0 0}, upload=497969]: Long tail cancellation due to optimal threshold Dec 29 07:28:37.489609460 (567h59m37.360692495s) [noise=true , address=77.90.31.11:4556 , piece_id=7VZCECY6U4FT4J3WMZMY4UKAAIVKP2F2ESAJLV4URD74H25FR5MA, seg_pos={0 0}, upload=497969]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 07:28:37.489619539 (567h59m37.360702584s) [noise=true , address=77.90.31.11:4556 , piece_id=7VZCECY6U4FT4J3WMZMY4UKAAIVKP2F2ESAJLV4URD74H25FR5MA, seg_pos={0 0}, upload=497969]: Long tail cancellation due to optimal threshold Dec 29 07:28:37.489680545 (567h59m37.36076358s) [seg_pos={0 0}, upload=497969]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:28:37.489835229 (567h59m37.360918264s) [upload=497969]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:28:37.489842272 (567h59m37.360925307s) [upload=497969]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:28:37.789935020 (567h59m37.661018055s) [upload=497969]: Done uploading segments... Dec 29 07:28:37.789953025 (567h59m37.66103606s) [upload=497969]: Upload finished. err: <nil> Duration: 30.002997868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X......X...........................X.] duration: 39.055781011s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:28:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 39.055796732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X.......X......................................] duration: 3.484723279s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:28:26 UTC) orbiter warning: context deadline exceeded Duration: 3.484749529s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X......X...........................X] duration: 1m0.004023376s trace: https://jaeger.dev.storj.io/trace/4ee5a9598037154d FAIL (2025-12-29 07:28:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :19.602435722 (567h58m19.473518757s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603346977 (567h58m19.474430022s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.603784942 (567h58m19.474867977s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604224058 (567h58m19.475307103s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.604686899 (567h58m19.475769944s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605133370 (567h58m19.476216415s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.605600358 (567h58m19.476683404s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606087797 (567h58m19.477170832s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.606643876 (567h58m19.477726911s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.607264207 (567h58m19.478347252s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.608969474 (567h58m19.480052519s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.609064004 (567h58m19.480147049s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.610906874 (567h58m19.481989919s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.611091485 (567h58m19.48217453s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.612648630 (567h58m19.483731695s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.614313229 (567h58m19.485396274s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:19.618890706 (567h58m19.489973741s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Uploading piece... Dec 29 07:27:20.500709209 (567h58m20.371792244s) [noise=true , address=193.149.164.143:4702 , piece_id=CU4O4YE3WHFP2J37A36G7IWWCZIRVZQWLAODPN2WRJTTQBF435BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000 Dec 29 07:27:20.987619650 (567h58m20.858702685s) [upload=497951]: Next returned nil segment Dec 29 07:27:20.987637725 (567h58m20.85872077s) [upload=497951]: Waiting for error group managing segments... Dec 29 07:27:21.464519439 (567h58m21.335602484s) [noise=true , address=145.79.227.16:4943 , piece_id=UZ7H6YOYRBP77XEQRT6LMZQH3GKSH7JSYUVJY5VIST4V32O6HXEA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465023930 (567h58m21.336106975s) [noise=true , address=145.79.227.7:4325 , piece_id=6BKOFSNNFPPL5IZXUCPSV2BI655NKTQYF3TAMQGTG2HS4QXS6D6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.465933782 (567h58m21.337016827s) [noise=true , address=145.79.227.15:4908 , piece_id=VJXZJ6NFR7MLQ5JQTOUI5HK6HESXIB5S7PPZ5KJDJE5OV276VJBQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.486313007 (567h58m21.357396042s) [noise=true , address=145.79.227.17:4974 , piece_id=6GTHOJPFPNQH76WXTLO2F4GZRPVVNXAAIPDWKIFFTLS3WSXVSYVQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.667570613 (567h58m21.538653658s) [noise=true , address=77.90.31.17:4776 , piece_id=EOGCW2YW7IUNFYYH262JROU3SWXFAMZA3LFUP6ZPA435F5X5XVJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:21.704101083 (567h58m21.575184128s) [noise=true , address=77.90.31.12:4613 , piece_id=JZ4SRAR7Q6CSYORRJWQEOJXZWCDXNRZMOPMDJ7CYVZKF4I66CYXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.304700046 (567h58m22.175783081s) [noise=true , address=64.49.13.26:4946 , piece_id=3TFUDG6EGQSJ5M47BX5NYGGQ4FYAS4B3N6CJOU6RI53N7FBWBFUA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.378692520 (567h58m22.249775555s) [noise=true , address=145.79.227.44:4200 , piece_id=N2EBOZLH6BBE3ALGW3DICOWP2AJD2GUI2YMVU7PEN43S4ACN5TJA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.428184541 (567h58m22.299267576s) [noise=true , address=145.79.227.33:4880 , piece_id=GNIHETZCOOF2ZBS4RS7YKMYZ3OY6I3LBL7I5MR5XBGAQ7U7RGV6Q, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.479745131 (567h58m22.350828166s) [noise=true , address=145.79.227.34:4915 , piece_id=PHUUVR53CHD3RZSLM2BFFTLETSEPCZ3NYKD6MJ2X3QHJGC4I7OBA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.481285043 (567h58m22.352368078s) [noise=true , address=145.79.227.49:4354 , piece_id=FQFJIU6TC2MDKFVXDAFXTPKTR3ANCS3L2N5RCHLNQ7ITH5NQX64A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.487553700 (567h58m22.358636735s) [noise=true , address=64.49.13.27:4958 , piece_id=6CGJX5EVFUUFEDPN4Q6BHEKVQUYS5RNFVCDZI6Z3MQA24DLLWLXA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.520041172 (567h58m22.391124208s) [noise=true , address=145.79.227.35:4495 , piece_id=G2WV6WKTQEHK6G55IL33S2L3T5DJQOA5HL2OSRY3SMCS3BP2V7BQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532804095 (567h58m22.40388714s) [noise=true , address=145.79.227.30:4764 , piece_id=QCROK42AWMGMLALP3YRARTKHAE4L2WTC2PAZDHCJ63UQSVT4K2NQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: <nil> Dec 29 07:27:22.532827960 (567h58m22.403910995s) [seg_pos={0 0}, upload=497951]: Segment reached optimal threshold of 14 pieces. Dec 29 07:27:22.533116049 (567h58m22.404199094s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533137510 (567h58m22.404220565s) [noise=true , address=145.79.227.40:4064 , piece_id=RCHIUS222KICSPT5A3JSIVIWCX7HL2QG2GQRV55NH2WQZ5UPKGIA, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533164952 (567h58m22.404247997s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51608->64.49.13.169:4705: use of closed network connection Dec 29 07:27:22.533172727 (567h58m22.404255762s) [noise=true , address=64.49.13.169:4705 , piece_id=Y7EGGYLGETO3VLESY7EJUQMSH4LVTCFLZIGPKWZHJXUPKNR65PEQ, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533422022 (567h58m22.404505077s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx): protocol: expected piece hash; context canceled; EOF Dec 29 07:27:22.533431559 (567h58m22.404514604s) [noise=true , address=145.79.227.29:4973 , piece_id=2UADKGBGQEH7VXBGAPPVSVL5UIYM24DBHBOUP2E45HWNYZO77Y2A, seg_pos={0 0}, upload=497951]: Long tail cancellation due to optimal threshold Dec 29 07:27:22.533450165 (567h58m22.4045332s) [seg_pos={0 0}, upload=497951]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:27:22.533588819 (567h58m22.404671864s) [upload=497951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:27:22.533597145 (567h58m22.40468019s) [upload=497951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:27:22.831809904 (567h58m22.702892939s) [upload=497951]: Done uploading segments... Dec 29 07:27:22.831828619 (567h58m22.702911664s) [upload=497951]: Upload finished. err: <nil> Duration: 1m0.004034498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X.XXXXX.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X...] duration: 6.675795014s trace: https://jaeger.dev.storj.io/trace/21f08339b4a2d18b FAIL (2025-12-29 07:28:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6838511 (567h55m33.817921557s) [upload=497916]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:24:34.194924430 (567h55m34.066007475s) [seg_pos={0 0}, upload=497916]: Begin upload segment... Dec 29 07:24:34.194968754 (567h55m34.066051789s) [seg_pos={0 0}, upload=497916]: Done begin upload segment. Dec 29 07:24:34.194992580 (567h55m34.066075615s) [noise=true , address=77.90.31.15:4697 , piece_id=KQ3UFGKXUSN42GUAPSSK3OQKEBJRYBUSWZONVYOLLHXPSI7NC2ZQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195002098 (567h55m34.066085133s) [noise=true , address=45.91.116.12:4227 , piece_id=P4COCUH63LJARVJC3PHV6YOMLZBXSBM452WXP3ZKDR4N7FHGI7BA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195088582 (567h55m34.066171617s) [noise=true , address=77.90.31.19:4844 , piece_id=3TVSIZ3XPGXV342VU5ZIQH2BSRM7UKTBR77B3VJ4RKUOZOTUEBVA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195547897 (567h55m34.066630943s) [noise=true , address=145.79.227.46:4603 , piece_id=6Z6HLQXQ472JLTLBMT2FDN275PSPGZVGD2ZC3VEGQ3CPHWEO2ZAQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195826919 (567h55m34.066909954s) [noise=true , address=145.79.227.15:4873 , piece_id=CFDAAOJ7ZBSYUXDZNGRRH7F4ZZRLBECUS75OJJUE5P7SI6QKVZIA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.196092724 (567h55m34.067175759s) [noise=true , address=145.79.227.48:4302 , piece_id=PBI3JKQQRKA5UKIX6HWDLTL362WCZ6AYS4YF7ZYXKOJUZNUNEFUA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.196440006 (567h55m34.067523041s) [noise=true , address=145.79.227.51:4409 , piece_id=RG5PHP5P2WIJI7I22MC5REUY6BLEA2TWC32K3TP7OQYNDV7K2IZA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.196716793 (567h55m34.067799828s) [noise=true , address=145.79.227.25:4619 , piece_id=JAHWISQEDZZCBV2LVK5EOHXEOB7N3I42VHTH54D2GG2NC54LZM4Q, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.197024910 (567h55m34.068107955s) [noise=true , address=45.91.116.11:4118 , piece_id=JMBPJ577EC35XNXKPCKBGIDEID3ULLI5CSH25DXBUNQFPBWIYPXQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.197339048 (567h55m34.068422083s) [noise=true , address=145.79.227.16:4912 , piece_id=UKTJ3SEYNYAADDFATPRZNQZQX5AA46VZJ2FA7PEGL6RUZCCLIM7Q, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.198600389 (567h55m34.069683424s) [noise=true , address=145.79.227.7:4294 , piece_id=K3R7CU3N5DRXN7JLC7MYGFYZUNJY7W2KSIJMD3G7BKAWB33IR7JA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.199492719 (567h55m34.070575764s) [noise=true , address=145.79.227.5:4223 , piece_id=HXCUZ2D5RGZIJ7WCFPEMNTLJZOXBKYXH5YQFUMPBJ7Y5JSGDGVEA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.200590529 (567h55m34.071673564s) [noise=true , address=45.91.116.6:4976 , piece_id=VQF6KSO6ZQF23KIHAFFIEP4HPJQ2RALIP3GN36SZOA4TWPQW2WVQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.201701925 (567h55m34.07278496s) [noise=true , address=145.79.227.45:4231 , piece_id=44RQKIVIMUIRQL5FVRL2CSGEOHOZUAAG6UBRZAVAELTYXCVN7QVA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.202802080 (567h55m34.073885115s) [noise=true , address=145.79.227.13:4802 , piece_id=O7GEDWKHST43O7IPZMUPVP3X2UMT22TITJAR6BCOCE2KSBABXTJA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.204089662 (567h55m34.075172707s) [noise=true , address=45.91.116.10:4082 , piece_id=OHLWPIX5ICAJG2CTFILCV76ZDFIYJXUS7XOVYVVHLU7LBJ3KFPJQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:35.901965209 (567h55m35.773048264s) [upload=497916]: Next returned nil segment Dec 29 07:24:35.901981099 (567h55m35.773064144s) [upload=497916]: Waiting for error group managing segments... Dec 29 07:24:36.181959606 (567h55m36.053042641s) [noise=true , address=145.79.227.15:4873 , piece_id=CFDAAOJ7ZBSYUXDZNGRRH7F4ZZRLBECUS75OJJUE5P7SI6QKVZIA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.191062539 (567h55m36.062145584s) [noise=true , address=145.79.227.5:4223 , piece_id=HXCUZ2D5RGZIJ7WCFPEMNTLJZOXBKYXH5YQFUMPBJ7Y5JSGDGVEA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.191331011 (567h55m36.062414046s) [noise=true , address=145.79.227.13:4802 , piece_id=O7GEDWKHST43O7IPZMUPVP3X2UMT22TITJAR6BCOCE2KSBABXTJA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.191843657 (567h55m36.062926692s) [noise=true , address=145.79.227.46:4603 , piece_id=6Z6HLQXQ472JLTLBMT2FDN275PSPGZVGD2ZC3VEGQ3CPHWEO2ZAQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.193615201 (567h55m36.064698236s) [noise=true , address=145.79.227.16:4912 , piece_id=UKTJ3SEYNYAADDFATPRZNQZQX5AA46VZJ2FA7PEGL6RUZCCLIM7Q, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.196625582 (567h55m36.067708617s) [noise=true , address=145.79.227.7:4294 , piece_id=K3R7CU3N5DRXN7JLC7MYGFYZUNJY7W2KSIJMD3G7BKAWB33IR7JA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.205195200 (567h55m36.076278235s) [noise=true , address=145.79.227.48:4302 , piece_id=PBI3JKQQRKA5UKIX6HWDLTL362WCZ6AYS4YF7ZYXKOJUZNUNEFUA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.205463782 (567h55m36.076546817s) [noise=true , address=145.79.227.51:4409 , piece_id=RG5PHP5P2WIJI7I22MC5REUY6BLEA2TWC32K3TP7OQYNDV7K2IZA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.205906165 (567h55m36.0769892s) [noise=true , address=145.79.227.25:4619 , piece_id=JAHWISQEDZZCBV2LVK5EOHXEOB7N3I42VHTH54D2GG2NC54LZM4Q, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.213492460 (567h55m36.084575495s) [noise=true , address=145.79.227.45:4231 , piece_id=44RQKIVIMUIRQL5FVRL2CSGEOHOZUAAG6UBRZAVAELTYXCVN7QVA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.433432282 (567h55m36.304515318s) [noise=true , address=45.91.116.11:4118 , piece_id=JMBPJ577EC35XNXKPCKBGIDEID3ULLI5CSH25DXBUNQFPBWIYPXQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.469959868 (567h55m36.341042903s) [noise=true , address=45.91.116.6:4976 , piece_id=VQF6KSO6ZQF23KIHAFFIEP4HPJQ2RALIP3GN36SZOA4TWPQW2WVQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.555751640 (567h55m36.426834675s) [noise=true , address=45.91.116.10:4082 , piece_id=OHLWPIX5ICAJG2CTFILCV76ZDFIYJXUS7XOVYVVHLU7LBJ3KFPJQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.555771697 (567h55m36.426854732s) [seg_pos={0 0}, upload=497916]: Segment reached optimal threshold of 13 pieces. Dec 29 07:24:36.555940559 (567h55m36.427023594s) [noise=true , address=77.90.31.15:4697 , piece_id=KQ3UFGKXUSN42GUAPSSK3OQKEBJRYBUSWZONVYOLLHXPSI7NC2ZQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:52918->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:52918->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52918->77.90.31.15:4697: use of closed network connection Dec 29 07:24:36.555945959 (567h55m36.427028994s) [noise=true , address=77.90.31.15:4697 , piece_id=KQ3UFGKXUSN42GUAPSSK3OQKEBJRYBUSWZONVYOLLHXPSI7NC2ZQ, seg_pos={0 0}, upload=497916]: Long tail cancellation due to optimal threshold Dec 29 07:24:36.556037684 (567h55m36.427120719s) [noise=true , address=77.90.31.19:4844 , piece_id=3TVSIZ3XPGXV342VU5ZIQH2BSRM7UKTBR77B3VJ4RKUOZOTUEBVA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:54810->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:54810->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54810->77.90.31.19:4844: use of closed network connection Dec 29 07:24:36.556054867 (567h55m36.427137903s) [noise=true , address=77.90.31.19:4844 , piece_id=3TVSIZ3XPGXV342VU5ZIQH2BSRM7UKTBR77B3VJ4RKUOZOTUEBVA, seg_pos={0 0}, upload=497916]: Long tail cancellation due to optimal threshold Dec 29 07:24:36.556122706 (567h55m36.427205741s) [noise=true , address=45.91.116.12:4227 , piece_id=P4COCUH63LJARVJC3PHV6YOMLZBXSBM452WXP3ZKDR4N7FHGI7BA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:50174->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:50174->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50174->45.91.116.12:4227: use of closed network connection Dec 29 07:24:36.556131243 (567h55m36.427214278s) [noise=true , address=45.91.116.12:4227 , piece_id=P4COCUH63LJARVJC3PHV6YOMLZBXSBM452WXP3ZKDR4N7FHGI7BA, seg_pos={0 0}, upload=497916]: Long tail cancellation due to optimal threshold Dec 29 07:24:36.556147273 (567h55m36.427230308s) [seg_pos={0 0}, upload=497916]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:24:36.556231043 (567h55m36.427314078s) [upload=497916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:24:36.556237916 (567h55m36.427320961s) [upload=497916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:24:36.848280224 (567h55m36.719363269s) [upload=497916]: Done uploading segments... Dec 29 07:24:36.848298198 (567h55m36.719381233s) [upload=497916]: Upload finished. err: <nil> Duration:* 6.675828287s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X.......X.....................................] duration: 3.557358839s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:27:58 UTC) orbiter warning: context deadline exceeded Duration: 3.557383224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X......X...........................] duration: 39.870285886s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:27:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 39.870311906s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.X.XXXXX.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X..] duration: 5.889237934s trace: https://jaeger.dev.storj.io/trace/21f08339b4a2d18b FAIL (2025-12-29 07:27:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6838511 (567h55m33.817921557s) [upload=497916]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:24:34.194924430 (567h55m34.066007475s) [seg_pos={0 0}, upload=497916]: Begin upload segment... Dec 29 07:24:34.194968754 (567h55m34.066051789s) [seg_pos={0 0}, upload=497916]: Done begin upload segment. Dec 29 07:24:34.194992580 (567h55m34.066075615s) [noise=true , address=77.90.31.15:4697 , piece_id=KQ3UFGKXUSN42GUAPSSK3OQKEBJRYBUSWZONVYOLLHXPSI7NC2ZQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195002098 (567h55m34.066085133s) [noise=true , address=45.91.116.12:4227 , piece_id=P4COCUH63LJARVJC3PHV6YOMLZBXSBM452WXP3ZKDR4N7FHGI7BA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195088582 (567h55m34.066171617s) [noise=true , address=77.90.31.19:4844 , piece_id=3TVSIZ3XPGXV342VU5ZIQH2BSRM7UKTBR77B3VJ4RKUOZOTUEBVA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195547897 (567h55m34.066630943s) [noise=true , address=145.79.227.46:4603 , piece_id=6Z6HLQXQ472JLTLBMT2FDN275PSPGZVGD2ZC3VEGQ3CPHWEO2ZAQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195826919 (567h55m34.066909954s) [noise=true , address=145.79.227.15:4873 , piece_id=CFDAAOJ7ZBSYUXDZNGRRH7F4ZZRLBECUS75OJJUE5P7SI6QKVZIA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.196092724 (567h55m34.067175759s) [noise=true , address=145.79.227.48:4302 , piece_id=PBI3JKQQRKA5UKIX6HWDLTL362WCZ6AYS4YF7ZYXKOJUZNUNEFUA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.196440006 (567h55m34.067523041s) [noise=true , address=145.79.227.51:4409 , piece_id=RG5PHP5P2WIJI7I22MC5REUY6BLEA2TWC32K3TP7OQYNDV7K2IZA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.196716793 (567h55m34.067799828s) [noise=true , address=145.79.227.25:4619 , piece_id=JAHWISQEDZZCBV2LVK5EOHXEOB7N3I42VHTH54D2GG2NC54LZM4Q, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.197024910 (567h55m34.068107955s) [noise=true , address=45.91.116.11:4118 , piece_id=JMBPJ577EC35XNXKPCKBGIDEID3ULLI5CSH25DXBUNQFPBWIYPXQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.197339048 (567h55m34.068422083s) [noise=true , address=145.79.227.16:4912 , piece_id=UKTJ3SEYNYAADDFATPRZNQZQX5AA46VZJ2FA7PEGL6RUZCCLIM7Q, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.198600389 (567h55m34.069683424s) [noise=true , address=145.79.227.7:4294 , piece_id=K3R7CU3N5DRXN7JLC7MYGFYZUNJY7W2KSIJMD3G7BKAWB33IR7JA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.199492719 (567h55m34.070575764s) [noise=true , address=145.79.227.5:4223 , piece_id=HXCUZ2D5RGZIJ7WCFPEMNTLJZOXBKYXH5YQFUMPBJ7Y5JSGDGVEA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.200590529 (567h55m34.071673564s) [noise=true , address=45.91.116.6:4976 , piece_id=VQF6KSO6ZQF23KIHAFFIEP4HPJQ2RALIP3GN36SZOA4TWPQW2WVQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.201701925 (567h55m34.07278496s) [noise=true , address=145.79.227.45:4231 , piece_id=44RQKIVIMUIRQL5FVRL2CSGEOHOZUAAG6UBRZAVAELTYXCVN7QVA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.202802080 (567h55m34.073885115s) [noise=true , address=145.79.227.13:4802 , piece_id=O7GEDWKHST43O7IPZMUPVP3X2UMT22TITJAR6BCOCE2KSBABXTJA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.204089662 (567h55m34.075172707s) [noise=true , address=45.91.116.10:4082 , piece_id=OHLWPIX5ICAJG2CTFILCV76ZDFIYJXUS7XOVYVVHLU7LBJ3KFPJQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:35.901965209 (567h55m35.773048264s) [upload=497916]: Next returned nil segment Dec 29 07:24:35.901981099 (567h55m35.773064144s) [upload=497916]: Waiting for error group managing segments... Dec 29 07:24:36.181959606 (567h55m36.053042641s) [noise=true , address=145.79.227.15:4873 , piece_id=CFDAAOJ7ZBSYUXDZNGRRH7F4ZZRLBECUS75OJJUE5P7SI6QKVZIA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.191062539 (567h55m36.062145584s) [noise=true , address=145.79.227.5:4223 , piece_id=HXCUZ2D5RGZIJ7WCFPEMNTLJZOXBKYXH5YQFUMPBJ7Y5JSGDGVEA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.191331011 (567h55m36.062414046s) [noise=true , address=145.79.227.13:4802 , piece_id=O7GEDWKHST43O7IPZMUPVP3X2UMT22TITJAR6BCOCE2KSBABXTJA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.191843657 (567h55m36.062926692s) [noise=true , address=145.79.227.46:4603 , piece_id=6Z6HLQXQ472JLTLBMT2FDN275PSPGZVGD2ZC3VEGQ3CPHWEO2ZAQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.193615201 (567h55m36.064698236s) [noise=true , address=145.79.227.16:4912 , piece_id=UKTJ3SEYNYAADDFATPRZNQZQX5AA46VZJ2FA7PEGL6RUZCCLIM7Q, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.196625582 (567h55m36.067708617s) [noise=true , address=145.79.227.7:4294 , piece_id=K3R7CU3N5DRXN7JLC7MYGFYZUNJY7W2KSIJMD3G7BKAWB33IR7JA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.205195200 (567h55m36.076278235s) [noise=true , address=145.79.227.48:4302 , piece_id=PBI3JKQQRKA5UKIX6HWDLTL362WCZ6AYS4YF7ZYXKOJUZNUNEFUA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.205463782 (567h55m36.076546817s) [noise=true , address=145.79.227.51:4409 , piece_id=RG5PHP5P2WIJI7I22MC5REUY6BLEA2TWC32K3TP7OQYNDV7K2IZA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.205906165 (567h55m36.0769892s) [noise=true , address=145.79.227.25:4619 , piece_id=JAHWISQEDZZCBV2LVK5EOHXEOB7N3I42VHTH54D2GG2NC54LZM4Q, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.213492460 (567h55m36.084575495s) [noise=true , address=145.79.227.45:4231 , piece_id=44RQKIVIMUIRQL5FVRL2CSGEOHOZUAAG6UBRZAVAELTYXCVN7QVA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.433432282 (567h55m36.304515318s) [noise=true , address=45.91.116.11:4118 , piece_id=JMBPJ577EC35XNXKPCKBGIDEID3ULLI5CSH25DXBUNQFPBWIYPXQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.469959868 (567h55m36.341042903s) [noise=true , address=45.91.116.6:4976 , piece_id=VQF6KSO6ZQF23KIHAFFIEP4HPJQ2RALIP3GN36SZOA4TWPQW2WVQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.555751640 (567h55m36.426834675s) [noise=true , address=45.91.116.10:4082 , piece_id=OHLWPIX5ICAJG2CTFILCV76ZDFIYJXUS7XOVYVVHLU7LBJ3KFPJQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.555771697 (567h55m36.426854732s) [seg_pos={0 0}, upload=497916]: Segment reached optimal threshold of 13 pieces. Dec 29 07:24:36.555940559 (567h55m36.427023594s) [noise=true , address=77.90.31.15:4697 , piece_id=KQ3UFGKXUSN42GUAPSSK3OQKEBJRYBUSWZONVYOLLHXPSI7NC2ZQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:52918->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:52918->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52918->77.90.31.15:4697: use of closed network connection Dec 29 07:24:36.555945959 (567h55m36.427028994s) [noise=true , address=77.90.31.15:4697 , piece_id=KQ3UFGKXUSN42GUAPSSK3OQKEBJRYBUSWZONVYOLLHXPSI7NC2ZQ, seg_pos={0 0}, upload=497916]: Long tail cancellation due to optimal threshold Dec 29 07:24:36.556037684 (567h55m36.427120719s) [noise=true , address=77.90.31.19:4844 , piece_id=3TVSIZ3XPGXV342VU5ZIQH2BSRM7UKTBR77B3VJ4RKUOZOTUEBVA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:54810->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:54810->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54810->77.90.31.19:4844: use of closed network connection Dec 29 07:24:36.556054867 (567h55m36.427137903s) [noise=true , address=77.90.31.19:4844 , piece_id=3TVSIZ3XPGXV342VU5ZIQH2BSRM7UKTBR77B3VJ4RKUOZOTUEBVA, seg_pos={0 0}, upload=497916]: Long tail cancellation due to optimal threshold Dec 29 07:24:36.556122706 (567h55m36.427205741s) [noise=true , address=45.91.116.12:4227 , piece_id=P4COCUH63LJARVJC3PHV6YOMLZBXSBM452WXP3ZKDR4N7FHGI7BA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:50174->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:50174->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50174->45.91.116.12:4227: use of closed network connection Dec 29 07:24:36.556131243 (567h55m36.427214278s) [noise=true , address=45.91.116.12:4227 , piece_id=P4COCUH63LJARVJC3PHV6YOMLZBXSBM452WXP3ZKDR4N7FHGI7BA, seg_pos={0 0}, upload=497916]: Long tail cancellation due to optimal threshold Dec 29 07:24:36.556147273 (567h55m36.427230308s) [seg_pos={0 0}, upload=497916]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:24:36.556231043 (567h55m36.427314078s) [upload=497916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:24:36.556237916 (567h55m36.427320961s) [upload=497916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:24:36.848280224 (567h55m36.719363269s) [upload=497916]: Done uploading segments... Dec 29 07:24:36.848298198 (567h55m36.719381233s) [upload=497916]: Upload finished. err: <nil> Duration:* 5.889269834s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X.......X....................................] duration: 3.368154682s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:26:55 UTC) orbiter warning: context deadline exceeded Duration: 3.368225016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X......X..........................] duration: 39.590057064s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:26:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 39.590074016s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X.......X...................................] duration: 3.407989345s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:25:59 UTC) orbiter warning: context deadline exceeded Duration: 3.408017597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.X.X.XXXXX.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X.] duration: 5.965459964s trace: https://jaeger.dev.storj.io/trace/21f08339b4a2d18b FAIL (2025-12-29 07:25:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6838511 (567h55m33.817921557s) [upload=497916]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:24:34.194924430 (567h55m34.066007475s) [seg_pos={0 0}, upload=497916]: Begin upload segment... Dec 29 07:24:34.194968754 (567h55m34.066051789s) [seg_pos={0 0}, upload=497916]: Done begin upload segment. Dec 29 07:24:34.194992580 (567h55m34.066075615s) [noise=true , address=77.90.31.15:4697 , piece_id=KQ3UFGKXUSN42GUAPSSK3OQKEBJRYBUSWZONVYOLLHXPSI7NC2ZQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195002098 (567h55m34.066085133s) [noise=true , address=45.91.116.12:4227 , piece_id=P4COCUH63LJARVJC3PHV6YOMLZBXSBM452WXP3ZKDR4N7FHGI7BA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195088582 (567h55m34.066171617s) [noise=true , address=77.90.31.19:4844 , piece_id=3TVSIZ3XPGXV342VU5ZIQH2BSRM7UKTBR77B3VJ4RKUOZOTUEBVA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195547897 (567h55m34.066630943s) [noise=true , address=145.79.227.46:4603 , piece_id=6Z6HLQXQ472JLTLBMT2FDN275PSPGZVGD2ZC3VEGQ3CPHWEO2ZAQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195826919 (567h55m34.066909954s) [noise=true , address=145.79.227.15:4873 , piece_id=CFDAAOJ7ZBSYUXDZNGRRH7F4ZZRLBECUS75OJJUE5P7SI6QKVZIA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.196092724 (567h55m34.067175759s) [noise=true , address=145.79.227.48:4302 , piece_id=PBI3JKQQRKA5UKIX6HWDLTL362WCZ6AYS4YF7ZYXKOJUZNUNEFUA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.196440006 (567h55m34.067523041s) [noise=true , address=145.79.227.51:4409 , piece_id=RG5PHP5P2WIJI7I22MC5REUY6BLEA2TWC32K3TP7OQYNDV7K2IZA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.196716793 (567h55m34.067799828s) [noise=true , address=145.79.227.25:4619 , piece_id=JAHWISQEDZZCBV2LVK5EOHXEOB7N3I42VHTH54D2GG2NC54LZM4Q, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.197024910 (567h55m34.068107955s) [noise=true , address=45.91.116.11:4118 , piece_id=JMBPJ577EC35XNXKPCKBGIDEID3ULLI5CSH25DXBUNQFPBWIYPXQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.197339048 (567h55m34.068422083s) [noise=true , address=145.79.227.16:4912 , piece_id=UKTJ3SEYNYAADDFATPRZNQZQX5AA46VZJ2FA7PEGL6RUZCCLIM7Q, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.198600389 (567h55m34.069683424s) [noise=true , address=145.79.227.7:4294 , piece_id=K3R7CU3N5DRXN7JLC7MYGFYZUNJY7W2KSIJMD3G7BKAWB33IR7JA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.199492719 (567h55m34.070575764s) [noise=true , address=145.79.227.5:4223 , piece_id=HXCUZ2D5RGZIJ7WCFPEMNTLJZOXBKYXH5YQFUMPBJ7Y5JSGDGVEA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.200590529 (567h55m34.071673564s) [noise=true , address=45.91.116.6:4976 , piece_id=VQF6KSO6ZQF23KIHAFFIEP4HPJQ2RALIP3GN36SZOA4TWPQW2WVQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.201701925 (567h55m34.07278496s) [noise=true , address=145.79.227.45:4231 , piece_id=44RQKIVIMUIRQL5FVRL2CSGEOHOZUAAG6UBRZAVAELTYXCVN7QVA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.202802080 (567h55m34.073885115s) [noise=true , address=145.79.227.13:4802 , piece_id=O7GEDWKHST43O7IPZMUPVP3X2UMT22TITJAR6BCOCE2KSBABXTJA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.204089662 (567h55m34.075172707s) [noise=true , address=45.91.116.10:4082 , piece_id=OHLWPIX5ICAJG2CTFILCV76ZDFIYJXUS7XOVYVVHLU7LBJ3KFPJQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:35.901965209 (567h55m35.773048264s) [upload=497916]: Next returned nil segment Dec 29 07:24:35.901981099 (567h55m35.773064144s) [upload=497916]: Waiting for error group managing segments... Dec 29 07:24:36.181959606 (567h55m36.053042641s) [noise=true , address=145.79.227.15:4873 , piece_id=CFDAAOJ7ZBSYUXDZNGRRH7F4ZZRLBECUS75OJJUE5P7SI6QKVZIA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.191062539 (567h55m36.062145584s) [noise=true , address=145.79.227.5:4223 , piece_id=HXCUZ2D5RGZIJ7WCFPEMNTLJZOXBKYXH5YQFUMPBJ7Y5JSGDGVEA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.191331011 (567h55m36.062414046s) [noise=true , address=145.79.227.13:4802 , piece_id=O7GEDWKHST43O7IPZMUPVP3X2UMT22TITJAR6BCOCE2KSBABXTJA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.191843657 (567h55m36.062926692s) [noise=true , address=145.79.227.46:4603 , piece_id=6Z6HLQXQ472JLTLBMT2FDN275PSPGZVGD2ZC3VEGQ3CPHWEO2ZAQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.193615201 (567h55m36.064698236s) [noise=true , address=145.79.227.16:4912 , piece_id=UKTJ3SEYNYAADDFATPRZNQZQX5AA46VZJ2FA7PEGL6RUZCCLIM7Q, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.196625582 (567h55m36.067708617s) [noise=true , address=145.79.227.7:4294 , piece_id=K3R7CU3N5DRXN7JLC7MYGFYZUNJY7W2KSIJMD3G7BKAWB33IR7JA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.205195200 (567h55m36.076278235s) [noise=true , address=145.79.227.48:4302 , piece_id=PBI3JKQQRKA5UKIX6HWDLTL362WCZ6AYS4YF7ZYXKOJUZNUNEFUA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.205463782 (567h55m36.076546817s) [noise=true , address=145.79.227.51:4409 , piece_id=RG5PHP5P2WIJI7I22MC5REUY6BLEA2TWC32K3TP7OQYNDV7K2IZA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.205906165 (567h55m36.0769892s) [noise=true , address=145.79.227.25:4619 , piece_id=JAHWISQEDZZCBV2LVK5EOHXEOB7N3I42VHTH54D2GG2NC54LZM4Q, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.213492460 (567h55m36.084575495s) [noise=true , address=145.79.227.45:4231 , piece_id=44RQKIVIMUIRQL5FVRL2CSGEOHOZUAAG6UBRZAVAELTYXCVN7QVA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.433432282 (567h55m36.304515318s) [noise=true , address=45.91.116.11:4118 , piece_id=JMBPJ577EC35XNXKPCKBGIDEID3ULLI5CSH25DXBUNQFPBWIYPXQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.469959868 (567h55m36.341042903s) [noise=true , address=45.91.116.6:4976 , piece_id=VQF6KSO6ZQF23KIHAFFIEP4HPJQ2RALIP3GN36SZOA4TWPQW2WVQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.555751640 (567h55m36.426834675s) [noise=true , address=45.91.116.10:4082 , piece_id=OHLWPIX5ICAJG2CTFILCV76ZDFIYJXUS7XOVYVVHLU7LBJ3KFPJQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.555771697 (567h55m36.426854732s) [seg_pos={0 0}, upload=497916]: Segment reached optimal threshold of 13 pieces. Dec 29 07:24:36.555940559 (567h55m36.427023594s) [noise=true , address=77.90.31.15:4697 , piece_id=KQ3UFGKXUSN42GUAPSSK3OQKEBJRYBUSWZONVYOLLHXPSI7NC2ZQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:52918->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:52918->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52918->77.90.31.15:4697: use of closed network connection Dec 29 07:24:36.555945959 (567h55m36.427028994s) [noise=true , address=77.90.31.15:4697 , piece_id=KQ3UFGKXUSN42GUAPSSK3OQKEBJRYBUSWZONVYOLLHXPSI7NC2ZQ, seg_pos={0 0}, upload=497916]: Long tail cancellation due to optimal threshold Dec 29 07:24:36.556037684 (567h55m36.427120719s) [noise=true , address=77.90.31.19:4844 , piece_id=3TVSIZ3XPGXV342VU5ZIQH2BSRM7UKTBR77B3VJ4RKUOZOTUEBVA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:54810->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:54810->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54810->77.90.31.19:4844: use of closed network connection Dec 29 07:24:36.556054867 (567h55m36.427137903s) [noise=true , address=77.90.31.19:4844 , piece_id=3TVSIZ3XPGXV342VU5ZIQH2BSRM7UKTBR77B3VJ4RKUOZOTUEBVA, seg_pos={0 0}, upload=497916]: Long tail cancellation due to optimal threshold Dec 29 07:24:36.556122706 (567h55m36.427205741s) [noise=true , address=45.91.116.12:4227 , piece_id=P4COCUH63LJARVJC3PHV6YOMLZBXSBM452WXP3ZKDR4N7FHGI7BA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:50174->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:50174->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50174->45.91.116.12:4227: use of closed network connection Dec 29 07:24:36.556131243 (567h55m36.427214278s) [noise=true , address=45.91.116.12:4227 , piece_id=P4COCUH63LJARVJC3PHV6YOMLZBXSBM452WXP3ZKDR4N7FHGI7BA, seg_pos={0 0}, upload=497916]: Long tail cancellation due to optimal threshold Dec 29 07:24:36.556147273 (567h55m36.427230308s) [seg_pos={0 0}, upload=497916]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:24:36.556231043 (567h55m36.427314078s) [upload=497916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:24:36.556237916 (567h55m36.427320961s) [upload=497916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:24:36.848280224 (567h55m36.719363269s) [upload=497916]: Done uploading segments... Dec 29 07:24:36.848298198 (567h55m36.719381233s) [upload=497916]: Upload finished. err: <nil> Duration:* 5.965483259s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X......X.........................] duration: 29.781490999s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:25:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 29.78151861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X.......X..................................] duration: 3.546720922s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:25:24 UTC) orbiter warning: context deadline exceeded Duration: 3.546749817s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X.......X.................................] duration: 3.381782308s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:25:06 UTC) orbiter warning: context deadline exceeded Duration: 3.381812808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.X.X.XXXXX.X....XXX...X.XX.X..X..X.......X.X.......X.XX..X] duration: 30.005250567s trace: https://jaeger.dev.storj.io/trace/21f08339b4a2d18b FAIL (2025-12-29 07:25:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6838511 (567h55m33.817921557s) [upload=497916]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:24:34.194924430 (567h55m34.066007475s) [seg_pos={0 0}, upload=497916]: Begin upload segment... Dec 29 07:24:34.194968754 (567h55m34.066051789s) [seg_pos={0 0}, upload=497916]: Done begin upload segment. Dec 29 07:24:34.194992580 (567h55m34.066075615s) [noise=true , address=77.90.31.15:4697 , piece_id=KQ3UFGKXUSN42GUAPSSK3OQKEBJRYBUSWZONVYOLLHXPSI7NC2ZQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195002098 (567h55m34.066085133s) [noise=true , address=45.91.116.12:4227 , piece_id=P4COCUH63LJARVJC3PHV6YOMLZBXSBM452WXP3ZKDR4N7FHGI7BA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195088582 (567h55m34.066171617s) [noise=true , address=77.90.31.19:4844 , piece_id=3TVSIZ3XPGXV342VU5ZIQH2BSRM7UKTBR77B3VJ4RKUOZOTUEBVA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195547897 (567h55m34.066630943s) [noise=true , address=145.79.227.46:4603 , piece_id=6Z6HLQXQ472JLTLBMT2FDN275PSPGZVGD2ZC3VEGQ3CPHWEO2ZAQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.195826919 (567h55m34.066909954s) [noise=true , address=145.79.227.15:4873 , piece_id=CFDAAOJ7ZBSYUXDZNGRRH7F4ZZRLBECUS75OJJUE5P7SI6QKVZIA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.196092724 (567h55m34.067175759s) [noise=true , address=145.79.227.48:4302 , piece_id=PBI3JKQQRKA5UKIX6HWDLTL362WCZ6AYS4YF7ZYXKOJUZNUNEFUA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.196440006 (567h55m34.067523041s) [noise=true , address=145.79.227.51:4409 , piece_id=RG5PHP5P2WIJI7I22MC5REUY6BLEA2TWC32K3TP7OQYNDV7K2IZA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.196716793 (567h55m34.067799828s) [noise=true , address=145.79.227.25:4619 , piece_id=JAHWISQEDZZCBV2LVK5EOHXEOB7N3I42VHTH54D2GG2NC54LZM4Q, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.197024910 (567h55m34.068107955s) [noise=true , address=45.91.116.11:4118 , piece_id=JMBPJ577EC35XNXKPCKBGIDEID3ULLI5CSH25DXBUNQFPBWIYPXQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.197339048 (567h55m34.068422083s) [noise=true , address=145.79.227.16:4912 , piece_id=UKTJ3SEYNYAADDFATPRZNQZQX5AA46VZJ2FA7PEGL6RUZCCLIM7Q, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.198600389 (567h55m34.069683424s) [noise=true , address=145.79.227.7:4294 , piece_id=K3R7CU3N5DRXN7JLC7MYGFYZUNJY7W2KSIJMD3G7BKAWB33IR7JA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.199492719 (567h55m34.070575764s) [noise=true , address=145.79.227.5:4223 , piece_id=HXCUZ2D5RGZIJ7WCFPEMNTLJZOXBKYXH5YQFUMPBJ7Y5JSGDGVEA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.200590529 (567h55m34.071673564s) [noise=true , address=45.91.116.6:4976 , piece_id=VQF6KSO6ZQF23KIHAFFIEP4HPJQ2RALIP3GN36SZOA4TWPQW2WVQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.201701925 (567h55m34.07278496s) [noise=true , address=145.79.227.45:4231 , piece_id=44RQKIVIMUIRQL5FVRL2CSGEOHOZUAAG6UBRZAVAELTYXCVN7QVA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.202802080 (567h55m34.073885115s) [noise=true , address=145.79.227.13:4802 , piece_id=O7GEDWKHST43O7IPZMUPVP3X2UMT22TITJAR6BCOCE2KSBABXTJA, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:34.204089662 (567h55m34.075172707s) [noise=true , address=45.91.116.10:4082 , piece_id=OHLWPIX5ICAJG2CTFILCV76ZDFIYJXUS7XOVYVVHLU7LBJ3KFPJQ, seg_pos={0 0}, upload=497916]: Uploading piece... Dec 29 07:24:35.901965209 (567h55m35.773048264s) [upload=497916]: Next returned nil segment Dec 29 07:24:35.901981099 (567h55m35.773064144s) [upload=497916]: Waiting for error group managing segments... Dec 29 07:24:36.181959606 (567h55m36.053042641s) [noise=true , address=145.79.227.15:4873 , piece_id=CFDAAOJ7ZBSYUXDZNGRRH7F4ZZRLBECUS75OJJUE5P7SI6QKVZIA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.191062539 (567h55m36.062145584s) [noise=true , address=145.79.227.5:4223 , piece_id=HXCUZ2D5RGZIJ7WCFPEMNTLJZOXBKYXH5YQFUMPBJ7Y5JSGDGVEA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.191331011 (567h55m36.062414046s) [noise=true , address=145.79.227.13:4802 , piece_id=O7GEDWKHST43O7IPZMUPVP3X2UMT22TITJAR6BCOCE2KSBABXTJA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.191843657 (567h55m36.062926692s) [noise=true , address=145.79.227.46:4603 , piece_id=6Z6HLQXQ472JLTLBMT2FDN275PSPGZVGD2ZC3VEGQ3CPHWEO2ZAQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.193615201 (567h55m36.064698236s) [noise=true , address=145.79.227.16:4912 , piece_id=UKTJ3SEYNYAADDFATPRZNQZQX5AA46VZJ2FA7PEGL6RUZCCLIM7Q, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.196625582 (567h55m36.067708617s) [noise=true , address=145.79.227.7:4294 , piece_id=K3R7CU3N5DRXN7JLC7MYGFYZUNJY7W2KSIJMD3G7BKAWB33IR7JA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.205195200 (567h55m36.076278235s) [noise=true , address=145.79.227.48:4302 , piece_id=PBI3JKQQRKA5UKIX6HWDLTL362WCZ6AYS4YF7ZYXKOJUZNUNEFUA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.205463782 (567h55m36.076546817s) [noise=true , address=145.79.227.51:4409 , piece_id=RG5PHP5P2WIJI7I22MC5REUY6BLEA2TWC32K3TP7OQYNDV7K2IZA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.205906165 (567h55m36.0769892s) [noise=true , address=145.79.227.25:4619 , piece_id=JAHWISQEDZZCBV2LVK5EOHXEOB7N3I42VHTH54D2GG2NC54LZM4Q, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.213492460 (567h55m36.084575495s) [noise=true , address=145.79.227.45:4231 , piece_id=44RQKIVIMUIRQL5FVRL2CSGEOHOZUAAG6UBRZAVAELTYXCVN7QVA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.433432282 (567h55m36.304515318s) [noise=true , address=45.91.116.11:4118 , piece_id=JMBPJ577EC35XNXKPCKBGIDEID3ULLI5CSH25DXBUNQFPBWIYPXQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.469959868 (567h55m36.341042903s) [noise=true , address=45.91.116.6:4976 , piece_id=VQF6KSO6ZQF23KIHAFFIEP4HPJQ2RALIP3GN36SZOA4TWPQW2WVQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.555751640 (567h55m36.426834675s) [noise=true , address=45.91.116.10:4082 , piece_id=OHLWPIX5ICAJG2CTFILCV76ZDFIYJXUS7XOVYVVHLU7LBJ3KFPJQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: <nil> Dec 29 07:24:36.555771697 (567h55m36.426854732s) [seg_pos={0 0}, upload=497916]: Segment reached optimal threshold of 13 pieces. Dec 29 07:24:36.555940559 (567h55m36.427023594s) [noise=true , address=77.90.31.15:4697 , piece_id=KQ3UFGKXUSN42GUAPSSK3OQKEBJRYBUSWZONVYOLLHXPSI7NC2ZQ, seg_pos={0 0}, upload=497916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:52918->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:52918->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52918->77.90.31.15:4697: use of closed network connection Dec 29 07:24:36.555945959 (567h55m36.427028994s) [noise=true , address=77.90.31.15:4697 , piece_id=KQ3UFGKXUSN42GUAPSSK3OQKEBJRYBUSWZONVYOLLHXPSI7NC2ZQ, seg_pos={0 0}, upload=497916]: Long tail cancellation due to optimal threshold Dec 29 07:24:36.556037684 (567h55m36.427120719s) [noise=true , address=77.90.31.19:4844 , piece_id=3TVSIZ3XPGXV342VU5ZIQH2BSRM7UKTBR77B3VJ4RKUOZOTUEBVA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:54810->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:54810->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54810->77.90.31.19:4844: use of closed network connection Dec 29 07:24:36.556054867 (567h55m36.427137903s) [noise=true , address=77.90.31.19:4844 , piece_id=3TVSIZ3XPGXV342VU5ZIQH2BSRM7UKTBR77B3VJ4RKUOZOTUEBVA, seg_pos={0 0}, upload=497916]: Long tail cancellation due to optimal threshold Dec 29 07:24:36.556122706 (567h55m36.427205741s) [noise=true , address=45.91.116.12:4227 , piece_id=P4COCUH63LJARVJC3PHV6YOMLZBXSBM452WXP3ZKDR4N7FHGI7BA, seg_pos={0 0}, upload=497916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:50174->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:50174->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50174->45.91.116.12:4227: use of closed network connection Dec 29 07:24:36.556131243 (567h55m36.427214278s) [noise=true , address=45.91.116.12:4227 , piece_id=P4COCUH63LJARVJC3PHV6YOMLZBXSBM452WXP3ZKDR4N7FHGI7BA, seg_pos={0 0}, upload=497916]: Long tail cancellation due to optimal threshold Dec 29 07:24:36.556147273 (567h55m36.427230308s) [seg_pos={0 0}, upload=497916]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:24:36.556231043 (567h55m36.427314078s) [upload=497916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:24:36.556237916 (567h55m36.427320961s) [upload=497916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:24:36.848280224 (567h55m36.719363269s) [upload=497916]: Done uploading segments... Dec 29 07:24:36.848298198 (567h55m36.719381233s) [upload=497916]: Upload finished. err: <nil> Duration:* 30.005277288s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X......X........................] duration: 41.987415328s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:24:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 41.987438732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X.......X................................] duration: 3.510451621s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:24:22 UTC) orbiter warning: context deadline exceeded Duration: 3.510473433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X......X.......................] duration: 41.032208448s trace: https://jaeger.dev.storj.io/trace/4b9db9f5162599b FAIL (2025-12-29 07:23:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 05:36.282272176 (567h36m36.153355211s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.282885994 (567h36m36.153969039s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283115281 (567h36m36.154198316s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.283585776 (567h36m36.154668811s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284065791 (567h36m36.155148826s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284097291 (567h36m36.155180326s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284509426 (567h36m36.155592471s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284557206 (567h36m36.155640241s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.284996503 (567h36m36.156079538s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285424358 (567h36m36.156507393s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.285929719 (567h36m36.157012754s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286287029 (567h36m36.157370065s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.286756664 (567h36m36.157839709s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287189979 (567h36m36.158273024s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287374611 (567h36m36.158457656s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.287753241 (567h36m36.158836276s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:36.288103979 (567h36m36.159187024s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Uploading piece... Dec 29 07:05:37.647815382 (567h36m37.518898417s) [upload=497636]: Next returned nil segment Dec 29 07:05:37.647828978 (567h36m37.518912013s) [upload=497636]: Waiting for error group managing segments... Dec 29 07:05:38.119138437 (567h36m37.990221472s) [noise=true , address=145.79.227.18:4018 , piece_id=HFCANVEDV73RLA6QFMQDHQA67ZT47QR3X6EJC6D73USZKDMBNCMQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.134693255 (567h36m38.00577629s) [noise=true , address=145.79.227.2:4139 , piece_id=KSLZIQDGQPPQXPD7VVFD436FXV36JADHCDIAYA3PWX6JDD3JHINQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.142716544 (567h36m38.013799579s) [noise=true , address=145.79.227.4:4213 , piece_id=BRWGJBCZFLYW5WN4QLZL2ZY4OAHQRA7O6A6HGBRRUTOVYHVXTPUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.145194913 (567h36m38.016277958s) [noise=true , address=145.79.227.12:4776 , piece_id=W37UWT2MCNMDKU6DNCSDUBGLQUKN7ASM67UCYPZEOH6KYTP47DEA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148336655 (567h36m38.0194197s) [noise=true , address=145.79.227.13:4831 , piece_id=ESUPQCOANQS4U4TOGVEEKTT7IZ4YRFKIC3WP4K4DTV7C5O6GBNDQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.148348738 (567h36m38.019431773s) [noise=true , address=145.79.227.43:4191 , piece_id=LGKGW2HD4OOWH2VBPS5ZJR6FSXRD5B37KLALAPKWIHIN6DDAI2RQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.302575317 (567h36m38.173658362s) [noise=true , address=77.90.31.18:4833 , piece_id=DKMXO7MV27GLZPH3E664WVT6OAFSSSQ44V73XDEJBRFDM5OO4XJA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.437051158 (567h36m38.308134193s) [noise=true , address=77.90.31.17:4782 , piece_id=GZ27JL2IEQYTZOIW7RLUSPRL2KLFJZ25OTQRXWVFCDKPYSEIGL2A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.440444099 (567h36m38.311527144s) [noise=true , address=77.90.31.15:4710 , piece_id=HUFQXYY3PTAT45U6KOFOBQHMZAMQQZDSHVSOMBF2O275YPTADBKA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:38.960891845 (567h36m38.83197488s) [noise=true , address=193.149.164.143:4699 , piece_id=5YJWK7PZSHXVVDJD62MMG3C2Q7XH22J5GMA552DZEB65EH63QBFA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.060197187 (567h36m38.931280222s) [noise=true , address=145.79.227.30:4777 , piece_id=YF5OHE2EG742KUFOU73MLZUZZIJKBMEYIW32OLNMO77J5SWAIEPQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.062129346 (567h36m38.933212381s) [noise=true , address=145.79.227.44:4203 , piece_id=CECRYJAXFJJSPZW4QHF32NPSNS7TO4YJAKSHNTI7FKFBAYI7PIYA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.064628044 (567h36m38.935711079s) [noise=true , address=145.79.227.40:4083 , piece_id=OLXISQMWSLDC64B3VNCLNM5LMCCFCAAHVOUS4JYFSLZX2QF2VR6A, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105333803 (567h36m38.976416838s) [noise=true , address=145.79.227.33:4889 , piece_id=FENXBO5C5U5ETM6GEBTBBHLHW5JDM4KZENPVG7METFBABB76HHQQ, seg_pos={0 0}, upload=497636]: Done uploading piece. err: <nil> Dec 29 07:05:39.105354122 (567h36m38.976437167s) [seg_pos={0 0}, upload=497636]: Segment reached optimal threshold of 14 pieces. Dec 29 07:05:39.105525809 (567h36m38.976608844s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105531981 (567h36m38.976615016s) [noise=true , address=77.90.31.11:4559 , piece_id=AA5WFQCSVMEXV5VZ7LHU5345ULZXUDJMJXC4VK24DT5EGJOVDL2Q, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105567749 (567h36m38.976650794s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105577878 (567h36m38.976660933s) [noise=true , address=145.79.227.49:4341 , piece_id=LRL5LILD3EGUT7C4G6RZ5QFDIFNWGVLHDPNSBW26KK4HAGC3QTQA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105634376 (567h36m38.976717411s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105639425 (567h36m38.97672246s) [noise=true , address=77.90.31.16:4750 , piece_id=ZY6LCIHKRH6AXAPMMJ2PQ4MREW6AH2TAPTDLOFYZEBSJBBDZ2XUA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105740909 (567h36m38.976823954s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP): protocol: expected piece hash; context canceled; EOF Dec 29 07:05:39.105748363 (567h36m38.976831408s) [noise=true , address=77.90.31.21:4885 , piece_id=5ZO4N67SO52HZUVWM7SX6WRTT63SPJGJZL6CGX2NPMRZ7TJXBCLA, seg_pos={0 0}, upload=497636]: Long tail cancellation due to optimal threshold Dec 29 07:05:39.105782979 (567h36m38.976866024s) [seg_pos={0 0}, upload=497636]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 07:05:39.105891255 (567h36m38.97697429s) [upload=497636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:05:39.105898017 (567h36m38.976981052s) [upload=497636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:05:39.397755485 (567h36m39.26883853s) [upload=497636]: Done uploading segments... Dec 29 07:05:39.397775683 (567h36m39.268858728s) [upload=497636]: Upload finished. err: <nil> Duration: 41.032232873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.X.X.X.XXXXX.X....XXX...X.XX.X..X..X.......X.X.......X.XX..] duration: 5.893325948s trace: https://jaeger.dev.storj.io/trace/6a4cd7f4d1543fdc FAIL (2025-12-29 07:23:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: QAWSDONJOIVLXEIQ5WVHS6ESDASIWMYBGHS5RG4TS3YBHGEQP6HA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: FUZVIYB7WVCC7FAMYEIYZ6UF5TELKNOEHMCWV4NTK5EJ4OTJ2HGA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: JTA32II7HLSSQDB6PALVE37O76WS2YF44WWNYFCS6G23ADH7PI4Q) context deadline exceeded ======LOGS====== Dec 29 07:22:04.673323868 (567h53m4.544406913s) [upload=497882]: Starting upload Dec 29 07:22:04.673434759 (567h53m4.544517794s) [upload=497882]: Done starting upload Dec 29 07:22:04.673492870 (567h53m4.544575915s) [upload=497882]: Uploading segments... Dec 29 07:22:04.673830613 (567h53m4.544913678s) [upload=497882]: Got next segment. Inline: false Dec 29 07:22:04.673893953 (567h53m4.544976998s) [upload=497882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 07:22:04.673900406 (567h53m4.544983471s) [upload=497882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 07:22:04.924666307 (567h53m4.795749352s) [seg_pos={0 0}, upload=497882]: Begin upload segment... Dec 29 07:22:04.924712545 (567h53m4.79579559s) [seg_pos={0 0}, upload=497882]: Done begin upload segment. Dec 29 07:22:04.924744676 (567h53m4.795827711s) [noise=true , address=145.79.227.4:4185 , piece_id=NDL4DKOWFDEZL3ORU4LMFCJZXQXJPJDMN7BBHJA55L2YEXROLUAA, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.924831030 (567h53m4.795914075s) [noise=true , address=45.91.116.8:4048 , piece_id=FUZVIYB7WVCC7FAMYEIYZ6UF5TELKNOEHMCWV4NTK5EJ4OTJ2HGA, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.925401457 (567h53m4.796484492s) [noise=true , address=77.90.31.10:4519 , piece_id=DWGR2GXJFC65FDCFC76ET3JEO7GXILLT3HCTYY7ZZG7PDA63ZYGA, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.926025034 (567h53m4.797108079s) [noise=true , address=145.79.227.41:4085 , piece_id=N6VDUAN7Z5JL7FYKOI7RYAOHNPLBPZ5WWSNJVL44EANDPB3P5ELA, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.926678629 (567h53m4.797761664s) [noise=true , address=45.91.116.10:4082 , piece_id=JTA32II7HLSSQDB6PALVE37O76WS2YF44WWNYFCS6G23ADH7PI4Q, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.927041259 (567h53m4.798124304s) [noise=true , address=145.79.227.38:4977 , piece_id=SLYN4FMSFMQ5N6M7PMPMS4PNNOBWZZPDDZHVXZ2GJUBZ22YIX2YA, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.927587179 (567h53m4.798670214s) [noise=true , address=77.90.31.14:4663 , piece_id=BBK5OIW2GSCVGNLSTAXYEQVX2MEA6LTA7FEVVKIQ3DF6B67C54EQ, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.927605603 (567h53m4.798688638s) [noise=true , address=45.91.116.14:4189 , piece_id=7ITUXOIK3VFFGORESULYY3AFFOKGVXDGSTBWH3LDJXWZC2J7N3DA, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.928063725 (567h53m4.799146771s) [noise=true , address=45.91.116.5:4298 , piece_id=QAWSDONJOIVLXEIQ5WVHS6ESDASIWMYBGHS5RG4TS3YBHGEQP6HA, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.928386110 (567h53m4.799469145s) [noise=true , address=45.91.116.11:4118 , piece_id=TF4RFOJ64NHJITOA26Y25OJKHX4DUAYK7RRXZSXMB2BIEROAN5KA, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.928510166 (567h53m4.799593211s) [noise=true , address=145.79.227.25:4619 , piece_id=DF2NDUB7GECQZPWSJT6M67MPTYVQ6GT2F33JA6UB3Z2WU5XJZ54Q, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.928878598 (567h53m4.799961633s) [noise=true , address=145.79.227.5:4223 , piece_id=Z7FK7UED5FZDA4KXMLILFSNVOMGOECUBXD3SG4GDN4HFDPCY3XEQ, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.929080412 (567h53m4.800163447s) [noise=true , address=145.79.227.24:4582 , piece_id=EUV32MHPHIA3JTFJJFENUYKBIAWKJLP4RUBI5JU6VP3RIQZGRNXQ, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.929307284 (567h53m4.800390319s) [noise=true , address=145.79.227.51:4409 , piece_id=SWEOCGU4WABRGROJ6DFNO5BWSDRYKM7GOJ4YPTL5CKZC4EGSANGA, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.929827524 (567h53m4.800910559s) [noise=true , address=145.79.227.3:4151 , piece_id=3W55OMLER63ASCV4UYEASAQJDTY4HPCIIMJ5WUWEMSTEQDJBVFEQ, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:04.929929268 (567h53m4.801012303s) [noise=true , address=45.91.116.12:4228 , piece_id=HUJRCDWCRD6KXXHG25XJHIFDPY7IARXWOTKNWBFZFNSSL4VTABXA, seg_pos={0 0}, upload=497882]: Uploading piece... Dec 29 07:22:06.622681648 (567h53m6.493764693s) [upload=497882]: Next returned nil segment Dec 29 07:22:06.622704961 (567h53m6.493787996s) [upload=497882]: Waiting for error group managing segments... Dec 29 07:22:06.895243021 (567h53m6.766326066s) [noise=true , address=145.79.227.4:4185 , piece_id=NDL4DKOWFDEZL3ORU4LMFCJZXQXJPJDMN7BBHJA55L2YEXROLUAA, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:06.915462723 (567h53m6.786545758s) [noise=true , address=145.79.227.25:4619 , piece_id=DF2NDUB7GECQZPWSJT6M67MPTYVQ6GT2F33JA6UB3Z2WU5XJZ54Q, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:06.916720758 (567h53m6.787803793s) [noise=true , address=145.79.227.5:4223 , piece_id=Z7FK7UED5FZDA4KXMLILFSNVOMGOECUBXD3SG4GDN4HFDPCY3XEQ, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:06.917999383 (567h53m6.789082418s) [noise=true , address=145.79.227.51:4409 , piece_id=SWEOCGU4WABRGROJ6DFNO5BWSDRYKM7GOJ4YPTL5CKZC4EGSANGA, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:06.920855321 (567h53m6.791938356s) [noise=true , address=145.79.227.38:4977 , piece_id=SLYN4FMSFMQ5N6M7PMPMS4PNNOBWZZPDDZHVXZ2GJUBZ22YIX2YA, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:06.924250035 (567h53m6.79533308s) [noise=true , address=145.79.227.41:4085 , piece_id=N6VDUAN7Z5JL7FYKOI7RYAOHNPLBPZ5WWSNJVL44EANDPB3P5ELA, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:06.931737102 (567h53m6.802820138s) [noise=true , address=145.79.227.3:4151 , piece_id=3W55OMLER63ASCV4UYEASAQJDTY4HPCIIMJ5WUWEMSTEQDJBVFEQ, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:06.942764991 (567h53m6.813848026s) [noise=true , address=145.79.227.24:4582 , piece_id=EUV32MHPHIA3JTFJJFENUYKBIAWKJLP4RUBI5JU6VP3RIQZGRNXQ, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:07.138599083 (567h53m7.009682118s) [noise=true , address=45.91.116.8:4048 , piece_id=FUZVIYB7WVCC7FAMYEIYZ6UF5TELKNOEHMCWV4NTK5EJ4OTJ2HGA, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:07.166181274 (567h53m7.037264309s) [noise=true , address=45.91.116.11:4118 , piece_id=TF4RFOJ64NHJITOA26Y25OJKHX4DUAYK7RRXZSXMB2BIEROAN5KA, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:07.188295964 (567h53m7.059378999s) [noise=true , address=45.91.116.5:4298 , piece_id=QAWSDONJOIVLXEIQ5WVHS6ESDASIWMYBGHS5RG4TS3YBHGEQP6HA, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:07.188836894 (567h53m7.059919939s) [noise=true , address=77.90.31.10:4519 , piece_id=DWGR2GXJFC65FDCFC76ET3JEO7GXILLT3HCTYY7ZZG7PDA63ZYGA, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:07.206732279 (567h53m7.077815324s) [noise=true , address=45.91.116.10:4082 , piece_id=JTA32II7HLSSQDB6PALVE37O76WS2YF44WWNYFCS6G23ADH7PI4Q, seg_pos={0 0}, upload=497882]: Done uploading piece. err: <nil> Dec 29 07:22:07.206760563 (567h53m7.077843608s) [seg_pos={0 0}, upload=497882]: Segment reached optimal threshold of 13 pieces. Dec 29 07:22:07.207214838 (567h53m7.078297883s) [noise=true , address=45.91.116.12:4228 , piece_id=HUJRCDWCRD6KXXHG25XJHIFDPY7IARXWOTKNWBFZFNSSL4VTABXA, seg_pos={0 0}, upload=497882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: read tcp 188.166.225.170:50240->45.91.116.12:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 07:22:07.207225259 (567h53m7.078308304s) [noise=true , address=45.91.116.12:4228 , piece_id=HUJRCDWCRD6KXXHG25XJHIFDPY7IARXWOTKNWBFZFNSSL4VTABXA, seg_pos={0 0}, upload=497882]: Long tail cancellation due to optimal threshold Dec 29 07:22:07.209491834 (567h53m7.080574879s) [noise=true , address=45.91.116.14:4189 , piece_id=7ITUXOIK3VFFGORESULYY3AFFOKGVXDGSTBWH3LDJXWZC2J7N3DA, seg_pos={0 0}, upload=497882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 29 07:22:07.209509317 (567h53m7.080592352s) [noise=true , address=45.91.116.14:4189 , piece_id=7ITUXOIK3VFFGORESULYY3AFFOKGVXDGSTBWH3LDJXWZC2J7N3DA, seg_pos={0 0}, upload=497882]: Long tail cancellation due to optimal threshold Dec 29 07:22:07.209665756 (567h53m7.080748801s) [noise=true , address=77.90.31.14:4663 , piece_id=BBK5OIW2GSCVGNLSTAXYEQVX2MEA6LTA7FEVVKIQ3DF6B67C54EQ, seg_pos={0 0}, upload=497882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 29 07:22:07.209673941 (567h53m7.080756986s) [noise=true , address=77.90.31.14:4663 , piece_id=BBK5OIW2GSCVGNLSTAXYEQVX2MEA6LTA7FEVVKIQ3DF6B67C54EQ, seg_pos={0 0}, upload=497882]: Long tail cancellation due to optimal threshold Dec 29 07:22:07.209698338 (567h53m7.080781403s) [seg_pos={0 0}, upload=497882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 07:22:07.209868903 (567h53m7.080951938s) [upload=497882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 07:22:07.209876948 (567h53m7.080959993s) [upload=497882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 07:22:07.521673732 (567h53m7.392756777s) [upload=497882]: Done uploading segments... Dec 29 07:22:07.521694862 (567h53m7.392777897s) [upload=497882]: Upload finished. err: <nil> Duration: 5.893364922s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X.......X...............................] duration: 3.539033124s trace: https://jaeger.dev.storj.io/trace/32b54f785e8cf2de FAIL (2025-12-29 07:23:18 UTC) orbiter warning: context deadline exceeded Duration: 3.539061357s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)