Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 34.286135278s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:18:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 34.286164444s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 251.382351ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:18:37 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 251.405486ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX.XX...X.X....X..X.X....X.......X.....................X....] duration: 6.419774623s trace: https://jaeger.dev.storj.io/trace/607cf963331dd581 FAIL (2025-12-29 20:18:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:14:59.343887631 (580h45m59.214970666s) [upload=509109]: Done starting upload Dec 29 20:14:59.343929450 (580h45m59.215012485s) [upload=509109]: Uploading segments... Dec 29 20:14:59.344178725 (580h45m59.21526176s) [upload=509109]: Got next segment. Inline: false Dec 29 20:14:59.344213341 (580h45m59.215296376s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:14:59.344218040 (580h45m59.215301075s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:14:59.620876250 (580h45m59.491959285s) [seg_pos={0 0}, upload=509109]: Begin upload segment... Dec 29 20:14:59.620932867 (580h45m59.492015912s) [seg_pos={0 0}, upload=509109]: Done begin upload segment. Dec 29 20:14:59.620975569 (580h45m59.492058604s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.620983133 (580h45m59.492066178s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621128750 (580h45m59.492211796s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621771063 (580h45m59.492854098s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622036208 (580h45m59.493119263s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622303207 (580h45m59.493386252s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622693610 (580h45m59.493776645s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623383705 (580h45m59.49446674s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623912161 (580h45m59.494995206s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624389991 (580h45m59.495473026s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624497966 (580h45m59.495581001s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624935829 (580h45m59.496018864s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624977129 (580h45m59.496060174s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625338717 (580h45m59.496421762s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625486658 (580h45m59.496569704s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625852676 (580h45m59.496935721s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:15:01.223830613 (580h46m1.094913658s) [upload=509109]: Next returned nil segment Dec 29 20:15:01.223865600 (580h46m1.094948645s) [upload=509109]: Waiting for error group managing segments... Dec 29 20:15:01.599999537 (580h46m1.471082582s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.606734191 (580h46m1.477817236s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.609790219 (580h46m1.480873265s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.613856723 (580h46m1.484939768s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.615434938 (580h46m1.486517973s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.617581145 (580h46m1.48866419s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.618039818 (580h46m1.489122863s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.624408906 (580h46m1.495491951s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.629708417 (580h46m1.500791482s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.654622637 (580h46m1.525705682s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.799321647 (580h46m1.670404682s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.902753888 (580h46m1.773836943s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167078173 (580h46m2.038161228s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167162874 (580h46m2.038245929s) [seg_pos={0 0}, upload=509109]: Segment reached optimal threshold of 13 pieces. Dec 29 20:15:02.170200207 (580h46m2.041283262s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42964->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 20:15:02.170215235 (580h46m2.04129829s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170672046 (580h46m2.041755101s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 20:15:02.170683507 (580h46m2.041766553s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170846298 (580h46m2.041929353s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection Dec 29 20:15:02.170859112 (580h46m2.041942157s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170884811 (580h46m2.041967856s) [seg_pos={0 0}, upload=509109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:15:02.171053743 (580h46m2.042136788s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:15:02.171062129 (580h46m2.042145184s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:15:02.455142652 (580h46m2.326225707s) [upload=509109]: Done uploading segments... Dec 29 20:15:02.455171597 (580h46m2.326254642s) [upload=509109]: Upload finished. err: <nil> Duration: 6.419816903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 18.460015153s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:18:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 18.460043527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXX.XX...X.X....X..X.X....X.......X.....................X...] duration: 7.681459059s trace: https://jaeger.dev.storj.io/trace/607cf963331dd581 FAIL (2025-12-29 20:18:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:14:59.343887631 (580h45m59.214970666s) [upload=509109]: Done starting upload Dec 29 20:14:59.343929450 (580h45m59.215012485s) [upload=509109]: Uploading segments... Dec 29 20:14:59.344178725 (580h45m59.21526176s) [upload=509109]: Got next segment. Inline: false Dec 29 20:14:59.344213341 (580h45m59.215296376s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:14:59.344218040 (580h45m59.215301075s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:14:59.620876250 (580h45m59.491959285s) [seg_pos={0 0}, upload=509109]: Begin upload segment... Dec 29 20:14:59.620932867 (580h45m59.492015912s) [seg_pos={0 0}, upload=509109]: Done begin upload segment. Dec 29 20:14:59.620975569 (580h45m59.492058604s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.620983133 (580h45m59.492066178s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621128750 (580h45m59.492211796s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621771063 (580h45m59.492854098s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622036208 (580h45m59.493119263s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622303207 (580h45m59.493386252s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622693610 (580h45m59.493776645s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623383705 (580h45m59.49446674s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623912161 (580h45m59.494995206s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624389991 (580h45m59.495473026s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624497966 (580h45m59.495581001s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624935829 (580h45m59.496018864s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624977129 (580h45m59.496060174s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625338717 (580h45m59.496421762s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625486658 (580h45m59.496569704s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625852676 (580h45m59.496935721s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:15:01.223830613 (580h46m1.094913658s) [upload=509109]: Next returned nil segment Dec 29 20:15:01.223865600 (580h46m1.094948645s) [upload=509109]: Waiting for error group managing segments... Dec 29 20:15:01.599999537 (580h46m1.471082582s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.606734191 (580h46m1.477817236s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.609790219 (580h46m1.480873265s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.613856723 (580h46m1.484939768s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.615434938 (580h46m1.486517973s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.617581145 (580h46m1.48866419s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.618039818 (580h46m1.489122863s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.624408906 (580h46m1.495491951s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.629708417 (580h46m1.500791482s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.654622637 (580h46m1.525705682s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.799321647 (580h46m1.670404682s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.902753888 (580h46m1.773836943s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167078173 (580h46m2.038161228s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167162874 (580h46m2.038245929s) [seg_pos={0 0}, upload=509109]: Segment reached optimal threshold of 13 pieces. Dec 29 20:15:02.170200207 (580h46m2.041283262s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42964->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 20:15:02.170215235 (580h46m2.04129829s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170672046 (580h46m2.041755101s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 20:15:02.170683507 (580h46m2.041766553s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170846298 (580h46m2.041929353s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection Dec 29 20:15:02.170859112 (580h46m2.041942157s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170884811 (580h46m2.041967856s) [seg_pos={0 0}, upload=509109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:15:02.171053743 (580h46m2.042136788s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:15:02.171062129 (580h46m2.042145184s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:15:02.455142652 (580h46m2.326225707s) [upload=509109]: Done uploading segments... Dec 29 20:15:02.455171597 (580h46m2.326254642s) [upload=509109]: Upload finished. err: <nil> Duration: 7.681493283s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 240.707614ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:17:58 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 240.727252ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 16.884058718s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:17:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 16.884090089s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 265.352725ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:17:06 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 265.395015ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XXX.XX...X.X....X..X.X....X.......X.....................X..] duration: 5.959096559s trace: https://jaeger.dev.storj.io/trace/607cf963331dd581 FAIL (2025-12-29 20:16:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:14:59.343887631 (580h45m59.214970666s) [upload=509109]: Done starting upload Dec 29 20:14:59.343929450 (580h45m59.215012485s) [upload=509109]: Uploading segments... Dec 29 20:14:59.344178725 (580h45m59.21526176s) [upload=509109]: Got next segment. Inline: false Dec 29 20:14:59.344213341 (580h45m59.215296376s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:14:59.344218040 (580h45m59.215301075s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:14:59.620876250 (580h45m59.491959285s) [seg_pos={0 0}, upload=509109]: Begin upload segment... Dec 29 20:14:59.620932867 (580h45m59.492015912s) [seg_pos={0 0}, upload=509109]: Done begin upload segment. Dec 29 20:14:59.620975569 (580h45m59.492058604s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.620983133 (580h45m59.492066178s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621128750 (580h45m59.492211796s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621771063 (580h45m59.492854098s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622036208 (580h45m59.493119263s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622303207 (580h45m59.493386252s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622693610 (580h45m59.493776645s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623383705 (580h45m59.49446674s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623912161 (580h45m59.494995206s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624389991 (580h45m59.495473026s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624497966 (580h45m59.495581001s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624935829 (580h45m59.496018864s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624977129 (580h45m59.496060174s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625338717 (580h45m59.496421762s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625486658 (580h45m59.496569704s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625852676 (580h45m59.496935721s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:15:01.223830613 (580h46m1.094913658s) [upload=509109]: Next returned nil segment Dec 29 20:15:01.223865600 (580h46m1.094948645s) [upload=509109]: Waiting for error group managing segments... Dec 29 20:15:01.599999537 (580h46m1.471082582s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.606734191 (580h46m1.477817236s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.609790219 (580h46m1.480873265s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.613856723 (580h46m1.484939768s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.615434938 (580h46m1.486517973s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.617581145 (580h46m1.48866419s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.618039818 (580h46m1.489122863s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.624408906 (580h46m1.495491951s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.629708417 (580h46m1.500791482s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.654622637 (580h46m1.525705682s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.799321647 (580h46m1.670404682s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.902753888 (580h46m1.773836943s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167078173 (580h46m2.038161228s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167162874 (580h46m2.038245929s) [seg_pos={0 0}, upload=509109]: Segment reached optimal threshold of 13 pieces. Dec 29 20:15:02.170200207 (580h46m2.041283262s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42964->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 20:15:02.170215235 (580h46m2.04129829s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170672046 (580h46m2.041755101s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 20:15:02.170683507 (580h46m2.041766553s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170846298 (580h46m2.041929353s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection Dec 29 20:15:02.170859112 (580h46m2.041942157s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170884811 (580h46m2.041967856s) [seg_pos={0 0}, upload=509109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:15:02.171053743 (580h46m2.042136788s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:15:02.171062129 (580h46m2.042145184s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:15:02.455142652 (580h46m2.326225707s) [upload=509109]: Done uploading segments... Dec 29 20:15:02.455171597 (580h46m2.326254642s) [upload=509109]: Upload finished. err: <nil> Duration: 5.959127787s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 15.409983061s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:16:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 15.410007838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 18.243300432s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:16:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 18.243328375s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 245.48481ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:16:12 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 245.509597ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XXX.XX...X.X....X..X.X....X.......X.....................X.] duration: 6.09757964s trace: https://jaeger.dev.storj.io/trace/607cf963331dd581 FAIL (2025-12-29 20:15:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:14:59.343887631 (580h45m59.214970666s) [upload=509109]: Done starting upload Dec 29 20:14:59.343929450 (580h45m59.215012485s) [upload=509109]: Uploading segments... Dec 29 20:14:59.344178725 (580h45m59.21526176s) [upload=509109]: Got next segment. Inline: false Dec 29 20:14:59.344213341 (580h45m59.215296376s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:14:59.344218040 (580h45m59.215301075s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:14:59.620876250 (580h45m59.491959285s) [seg_pos={0 0}, upload=509109]: Begin upload segment... Dec 29 20:14:59.620932867 (580h45m59.492015912s) [seg_pos={0 0}, upload=509109]: Done begin upload segment. Dec 29 20:14:59.620975569 (580h45m59.492058604s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.620983133 (580h45m59.492066178s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621128750 (580h45m59.492211796s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621771063 (580h45m59.492854098s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622036208 (580h45m59.493119263s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622303207 (580h45m59.493386252s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622693610 (580h45m59.493776645s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623383705 (580h45m59.49446674s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623912161 (580h45m59.494995206s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624389991 (580h45m59.495473026s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624497966 (580h45m59.495581001s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624935829 (580h45m59.496018864s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624977129 (580h45m59.496060174s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625338717 (580h45m59.496421762s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625486658 (580h45m59.496569704s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625852676 (580h45m59.496935721s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:15:01.223830613 (580h46m1.094913658s) [upload=509109]: Next returned nil segment Dec 29 20:15:01.223865600 (580h46m1.094948645s) [upload=509109]: Waiting for error group managing segments... Dec 29 20:15:01.599999537 (580h46m1.471082582s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.606734191 (580h46m1.477817236s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.609790219 (580h46m1.480873265s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.613856723 (580h46m1.484939768s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.615434938 (580h46m1.486517973s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.617581145 (580h46m1.48866419s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.618039818 (580h46m1.489122863s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.624408906 (580h46m1.495491951s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.629708417 (580h46m1.500791482s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.654622637 (580h46m1.525705682s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.799321647 (580h46m1.670404682s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.902753888 (580h46m1.773836943s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167078173 (580h46m2.038161228s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167162874 (580h46m2.038245929s) [seg_pos={0 0}, upload=509109]: Segment reached optimal threshold of 13 pieces. Dec 29 20:15:02.170200207 (580h46m2.041283262s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42964->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 20:15:02.170215235 (580h46m2.04129829s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170672046 (580h46m2.041755101s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 20:15:02.170683507 (580h46m2.041766553s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170846298 (580h46m2.041929353s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection Dec 29 20:15:02.170859112 (580h46m2.041942157s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170884811 (580h46m2.041967856s) [seg_pos={0 0}, upload=509109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:15:02.171053743 (580h46m2.042136788s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:15:02.171062129 (580h46m2.042145184s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:15:02.455142652 (580h46m2.326225707s) [upload=509109]: Done uploading segments... Dec 29 20:15:02.455171597 (580h46m2.326254642s) [upload=509109]: Upload finished. err: <nil> Duration: 6.097605189s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..XXX.XX...X.X....X..X.X....X.......X.....................X] duration: 30.002515786s trace: https://jaeger.dev.storj.io/trace/607cf963331dd581 FAIL (2025-12-29 20:15:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 20:14:59.343887631 (580h45m59.214970666s) [upload=509109]: Done starting upload Dec 29 20:14:59.343929450 (580h45m59.215012485s) [upload=509109]: Uploading segments... Dec 29 20:14:59.344178725 (580h45m59.21526176s) [upload=509109]: Got next segment. Inline: false Dec 29 20:14:59.344213341 (580h45m59.215296376s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 20:14:59.344218040 (580h45m59.215301075s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 20:14:59.620876250 (580h45m59.491959285s) [seg_pos={0 0}, upload=509109]: Begin upload segment... Dec 29 20:14:59.620932867 (580h45m59.492015912s) [seg_pos={0 0}, upload=509109]: Done begin upload segment. Dec 29 20:14:59.620975569 (580h45m59.492058604s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.620983133 (580h45m59.492066178s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621128750 (580h45m59.492211796s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.621771063 (580h45m59.492854098s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622036208 (580h45m59.493119263s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622303207 (580h45m59.493386252s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.622693610 (580h45m59.493776645s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623383705 (580h45m59.49446674s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.623912161 (580h45m59.494995206s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624389991 (580h45m59.495473026s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624497966 (580h45m59.495581001s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624935829 (580h45m59.496018864s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.624977129 (580h45m59.496060174s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625338717 (580h45m59.496421762s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625486658 (580h45m59.496569704s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:14:59.625852676 (580h45m59.496935721s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Uploading piece... Dec 29 20:15:01.223830613 (580h46m1.094913658s) [upload=509109]: Next returned nil segment Dec 29 20:15:01.223865600 (580h46m1.094948645s) [upload=509109]: Waiting for error group managing segments... Dec 29 20:15:01.599999537 (580h46m1.471082582s) [noise=true , address=145.79.227.11:4731 , piece_id=46XM74D3UEXNZXHZYWFOYW6SHHWTQU2HA6Q2WYSJLH7OVC23JALQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.606734191 (580h46m1.477817236s) [noise=true , address=145.79.227.9:4366 , piece_id=ICYZM4QQZQJLS4T4KYM47NEMIKYD7PZWQ2U2NEFLAWLLPO6OFG3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.609790219 (580h46m1.480873265s) [noise=true , address=145.79.227.48:4303 , piece_id=UYEJZYH7QYLL6KGKEQBH6DEGEBMWZSM2OT7W4V6D4D3FQV4XUDZQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.613856723 (580h46m1.484939768s) [noise=true , address=145.79.227.51:4411 , piece_id=LYCY2Z7Z3BFR5XSM6JXFCL5LZAU56PUBVGHLVURZUTQTEZF7CLFQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.615434938 (580h46m1.486517973s) [noise=true , address=145.79.227.46:4602 , piece_id=ZJA75URLP6YW2ROXGLR4PONX2PXXWZNGGANAI37SREOPJ5S7OV3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.617581145 (580h46m1.48866419s) [noise=true , address=145.79.227.19:4078 , piece_id=NZLVPFPYJXZHMK2XSIMIDBR7CN4OWVWT5QGNLTK3STC5KBY7CMGA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.618039818 (580h46m1.489122863s) [noise=true , address=145.79.227.15:4875 , piece_id=RFIPM3MTYFN5PLV4NVNF7AE5OVA54ZIOR2PUY7S4674CN466ECBA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.624408906 (580h46m1.495491951s) [noise=true , address=145.79.227.36:4518 , piece_id=3DYS7R2VTEUGCDWBWFP2OZHYTMQGLK7OS3OFSCUBIKWSWHJ44ELQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.629708417 (580h46m1.500791482s) [noise=true , address=145.79.227.2:4115 , piece_id=J6ZAJU7FYPAFEI47EJBUUUCASWZY2PZZWABSA7WRLAQPYQMFTENQ, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.654622637 (580h46m1.525705682s) [noise=true , address=145.79.227.18:4006 , piece_id=QP3MYNJN7EGPAM7NB4E2ITEV6EPATVAIWGAPT3KNEUXEPPBNQVDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.799321647 (580h46m1.670404682s) [noise=true , address=77.90.31.13:4627 , piece_id=OVJXRA53CFJFZSRJ6H2MHZLO4RCLLC3TIGGYQN3ESNQFA3ESQD3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:01.902753888 (580h46m1.773836943s) [noise=true , address=45.91.116.7:4010 , piece_id=BO772OMWJBZDCBAX3O6ECC766FEHZVTOJHKJHDLCSXJWKEKIV6BA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167078173 (580h46m2.038161228s) [noise=true , address=45.91.116.9:4262 , piece_id=UV5XMT447DE6QTJYRYBD7466OPDQ3YQSIVQ4C2BSIO37OCZKGSHA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: <nil> Dec 29 20:15:02.167162874 (580h46m2.038245929s) [seg_pos={0 0}, upload=509109]: Segment reached optimal threshold of 13 pieces. Dec 29 20:15:02.170200207 (580h46m2.041283262s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42964->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 20:15:02.170215235 (580h46m2.04129829s) [noise=true , address=45.91.116.6:4975 , piece_id=YHNIUJWPFG4EC7FJYL43CCVFTKZ2GMKX5ANKEIHDMLAL2AGA7V3A, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170672046 (580h46m2.041755101s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 20:15:02.170683507 (580h46m2.041766553s) [noise=true , address=145.79.227.44:4194 , piece_id=NEQVB6ERVDONQCXWRFXXFPZ3EBSLRJJMWOZHSBCKZWFWZ5FJXTNA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170846298 (580h46m2.041929353s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56476->145.79.227.28:4725: use of closed network connection Dec 29 20:15:02.170859112 (580h46m2.041942157s) [noise=true , address=145.79.227.28:4725 , piece_id=U442UUBRULRA3AY6H7EP5G2XJFY47YYDMBWRCMQBM2BLCKZJBYDA, seg_pos={0 0}, upload=509109]: Long tail cancellation due to optimal threshold Dec 29 20:15:02.170884811 (580h46m2.041967856s) [seg_pos={0 0}, upload=509109]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 20:15:02.171053743 (580h46m2.042136788s) [upload=509109]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 20:15:02.171062129 (580h46m2.042145184s) [upload=509109]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 20:15:02.455142652 (580h46m2.326225707s) [upload=509109]: Done uploading segments... Dec 29 20:15:02.455171597 (580h46m2.326254642s) [upload=509109]: Upload finished. err: <nil> Duration: 30.002526306s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 248.475205ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:15:26 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 248.505021ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 20.939169449s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:15:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 20.939200459s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 242.38376ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:14:57 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 242.412404ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 15.697718455s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:14:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 15.697750405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX..XXX.XX...X.X....X..X.X....X.......X.....................] duration: 11.580295052s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:14:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 11.580325159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 28.61333328s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:14:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 28.613354469s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 242.714266ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:13:46 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 242.737991ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXX..XXX.XX...X.X....X..X.X....X.......X....................] duration: 6.283037977s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:13:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 6.28306654s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 26.9540672s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:13:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 26.95409301s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 250.964025ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:13:15 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 250.994713ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 17.765986418s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:12:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 17.766007639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXX..XXX.XX...X.X....X..X.X....X.......X...................] duration: 6.015116532s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:12:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 6.015146249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XXXX..XXX.XX...X.X....X..X.X....X.......X..................] duration: 10.384307514s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:12:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 10.384338964s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 709.974771ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:12:16 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 710.004037ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 15.410217977s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:12:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 15.410233947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 36.989197563s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:11:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 36.989219965s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 282.334869ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:11:34 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 282.360828ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XXXX..XXX.XX...X.X....X..X.X....X.......X.................] duration: 5.840141933s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:11:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 5.840173242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...XXXX..XXX.XX...X.X....X..X.X....X.......X................] duration: 26.738346062s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:10:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 26.738373344s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 250.344795ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:10:48 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 250.371136ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 33.040609312s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:10:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 33.040637465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 21.959257754s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:09:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 21.959289014s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 249.227067ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:09:43 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 249.271622ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....XXXX..XXX.XX...X.X....X..X.X....X.......X...............] duration: 6.104620137s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:09:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 6.104642951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....XXXX..XXX.XX...X.X....X..X.X....X.......X..............] duration: 6.61001939s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:08:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 6.610053896s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 258.431143ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:08:51 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 258.455349ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 13.551153391s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:08:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 13.551239354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......XXXX..XXX.XX...X.X....X..X.X....X.......X.............] duration: 14.301171691s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:08:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 14.301205163s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 255.843845ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:07:45 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 255.885185ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......XXXX..XXX.XX...X.X....X..X.X....X.......X............] duration: 5.740965073s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:07:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 5.740995581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..............X............................................] duration: 14.84377601s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:07:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 14.843827119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [........XXXX..XXX.XX...X.X....X..X.X....X.......X...........] duration: 5.956933979s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:07:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 5.956965992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..............X...........................................] duration: 29.506727546s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:06:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 29.506752234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X........XXXX..XXX.XX...X.X....X..X.X....X.......X..........] duration: 7.280499076s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:06:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 7.280536747s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 251.564037ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:06:33 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 251.59134ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X........XXXX..XXX.XX...X.X....X..X.X....X.......X.........] duration: 10.802534938s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:06:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 10.802553914s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)