Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 55.796946871s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:25:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 55.797019581s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.....X....X............XX........X..X......X.......XXX...X..] duration: 5.000997994s trace: https://jaeger.dev.storj.io/trace/77fa83e3c4f9fd80 FAIL (2025-12-26 12:25:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001023042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.X.X...X.X........X..XXXXX...X.X....X..X...X..XX.....XXX] duration: 30.004258735s trace: https://jaeger.dev.storj.io/trace/26841f27339b8a5 FAIL (2025-12-26 12:25:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 12:24:35.080916830 (500h55m34.951999875s) [upload=439635]: Starting upload Dec 26 12:24:35.080971635 (500h55m34.95205467s) [upload=439635]: Done starting upload Dec 26 12:24:35.081007403 (500h55m34.952090448s) [upload=439635]: Uploading segments... Dec 26 12:24:35.081188447 (500h55m34.952271492s) [upload=439635]: Got next segment. Inline: false Dec 26 12:24:35.081234977 (500h55m34.952318012s) [upload=439635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:24:35.081239956 (500h55m34.952323011s) [upload=439635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:24:35.325667789 (500h55m35.196750824s) [seg_pos={0 0}, upload=439635]: Begin upload segment... Dec 26 12:24:35.325729938 (500h55m35.196813023s) [seg_pos={0 0}, upload=439635]: Done begin upload segment. Dec 26 12:24:35.325828064 (500h55m35.196911099s) [noise=true , address=145.79.227.3:4151 , piece_id=64NDHUWNKYC2VUEQVSM5FHYRXDUZVIMICV5OYTEAZ7XUZEWMZAXA, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.325850116 (500h55m35.196933181s) [noise=true , address=145.79.227.4:4185 , piece_id=Y4TCHJM2HMXFSREPRM7CNRAJNFLKBMHAMPTK2GDDRBPUAOUJEYSQ, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.326294573 (500h55m35.197377608s) [noise=true , address=145.79.227.37:4565 , piece_id=YZEOYGNLPWHNHS4UJUHFOTHKNKZMI6NOPGSTNYMJPWAD6P7MHIHQ, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.326559597 (500h55m35.197642642s) [noise=true , address=45.91.116.9:4262 , piece_id=M5HJYJ47Y67O6PFMJA7DVG4GGRLWCW3UKD4TYHQVPUT3RWUDNOMQ, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.326962084 (500h55m35.198045119s) [noise=true , address=45.91.116.5:4299 , piece_id=XYGPHPG635SWQC4FK4BJ57CP7NS7FQ5SNYDIE6TXUZ4YEHXWOJ6Q, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.327296601 (500h55m35.198379646s) [noise=true , address=145.79.227.39:4013 , piece_id=JONFC4DXSMQ4M6EHSI2WMWQPVMTV7ECPWPLZVVWWRZIZPYXGMEIA, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.328428676 (500h55m35.199511711s) [noise=true , address=145.79.227.2:4115 , piece_id=RV3FVIP33ZK3TA3K5FQ267W6VSCAKFVPVKQLTLZGFRE74SHS762Q, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.328527615 (500h55m35.19961065s) [noise=true , address=145.79.227.35:4483 , piece_id=GDEUR2H62FYDW7WDYGMHG2CXDM5PLENJ3GOTERZWALN5Z5J53T6Q, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.328868935 (500h55m35.19995197s) [noise=true , address=145.79.227.36:4517 , piece_id=F5SEVZ2ST4BI3KLSYBS6EEEPJYF4RR4TBC4ADJYDB6GGMDIE5VBA, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.329267613 (500h55m35.200350649s) [noise=true , address=45.91.116.13:4154 , piece_id=X3EDJQ2O2BJJIXII57NHATPKJU64KIQXYSJGNSRSVKW2OPWVSA6Q, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.329334611 (500h55m35.200417646s) [noise=true , address=145.79.227.41:4087 , piece_id=5TRI2KKJDFFOLFKPUVJ3AC4WOEQIUGEZW7OT75JASAAH4BLAPC7Q, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.329681993 (500h55m35.200765038s) [noise=true , address=45.91.116.6:4975 , piece_id=3KNMPGK2IOK6CGJUJ2SK223KWHRCKXM6MSPD2YHMTOZKT377RRFA, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.329974731 (500h55m35.201057776s) [noise=true , address=45.91.116.8:4046 , piece_id=XWUYYNVKZE2HMELUMRJGKRCTQ2N3N5GMCBQ7QCSIBS75IMRR5WBA, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.330288779 (500h55m35.201371814s) [noise=true , address=145.79.227.15:4874 , piece_id=KNOHOB24HTFR7AWRFXPQMMY7JKBOWQ2HZIVWXIMJ5QBLVBOHRDVQ, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.330633765 (500h55m35.2017168s) [noise=true , address=45.91.116.7:4011 , piece_id=7EYK2CUR3CZCCAPZLJNVPK7SKSA33NSUOGS647CGGPJWTNTPONZQ, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:35.330878131 (500h55m35.201961166s) [noise=true , address=145.79.227.13:4802 , piece_id=O3K3WIQALVWQTK4ZZPCBKEF3WPUUDJUWCQQJMTANLXFX4URWKS5A, seg_pos={0 0}, upload=439635]: Uploading piece... Dec 26 12:24:36.948579116 (500h55m36.819662161s) [upload=439635]: Next returned nil segment Dec 26 12:24:36.948597972 (500h55m36.819681017s) [upload=439635]: Waiting for error group managing segments... Dec 26 12:24:37.319676670 (500h55m37.190759715s) [noise=true , address=145.79.227.2:4115 , piece_id=RV3FVIP33ZK3TA3K5FQ267W6VSCAKFVPVKQLTLZGFRE74SHS762Q, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.328946722 (500h55m37.200029767s) [noise=true , address=145.79.227.35:4483 , piece_id=GDEUR2H62FYDW7WDYGMHG2CXDM5PLENJ3GOTERZWALN5Z5J53T6Q, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.330351647 (500h55m37.201434692s) [noise=true , address=145.79.227.37:4565 , piece_id=YZEOYGNLPWHNHS4UJUHFOTHKNKZMI6NOPGSTNYMJPWAD6P7MHIHQ, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.335216321 (500h55m37.206299366s) [noise=true , address=145.79.227.13:4802 , piece_id=O3K3WIQALVWQTK4ZZPCBKEF3WPUUDJUWCQQJMTANLXFX4URWKS5A, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.347271996 (500h55m37.218355042s) [noise=true , address=145.79.227.39:4013 , piece_id=JONFC4DXSMQ4M6EHSI2WMWQPVMTV7ECPWPLZVVWWRZIZPYXGMEIA, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.349304036 (500h55m37.220387092s) [noise=true , address=145.79.227.36:4517 , piece_id=F5SEVZ2ST4BI3KLSYBS6EEEPJYF4RR4TBC4ADJYDB6GGMDIE5VBA, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.354319457 (500h55m37.225402502s) [noise=true , address=145.79.227.3:4151 , piece_id=64NDHUWNKYC2VUEQVSM5FHYRXDUZVIMICV5OYTEAZ7XUZEWMZAXA, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.355279945 (500h55m37.22636299s) [noise=true , address=145.79.227.41:4087 , piece_id=5TRI2KKJDFFOLFKPUVJ3AC4WOEQIUGEZW7OT75JASAAH4BLAPC7Q, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.357500313 (500h55m37.228583358s) [noise=true , address=145.79.227.15:4874 , piece_id=KNOHOB24HTFR7AWRFXPQMMY7JKBOWQ2HZIVWXIMJ5QBLVBOHRDVQ, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.360044578 (500h55m37.231127623s) [noise=true , address=145.79.227.4:4185 , piece_id=Y4TCHJM2HMXFSREPRM7CNRAJNFLKBMHAMPTK2GDDRBPUAOUJEYSQ, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.594160856 (500h55m37.465243901s) [noise=true , address=45.91.116.6:4975 , piece_id=3KNMPGK2IOK6CGJUJ2SK223KWHRCKXM6MSPD2YHMTOZKT377RRFA, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.602441213 (500h55m37.473524259s) [noise=true , address=45.91.116.7:4011 , piece_id=7EYK2CUR3CZCCAPZLJNVPK7SKSA33NSUOGS647CGGPJWTNTPONZQ, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.693906670 (500h55m37.564989705s) [noise=true , address=45.91.116.9:4262 , piece_id=M5HJYJ47Y67O6PFMJA7DVG4GGRLWCW3UKD4TYHQVPUT3RWUDNOMQ, seg_pos={0 0}, upload=439635]: Done uploading piece. err: <nil> Dec 26 12:24:37.693945834 (500h55m37.56502888s) [seg_pos={0 0}, upload=439635]: Segment reached optimal threshold of 13 pieces. Dec 26 12:24:37.694242559 (500h55m37.565325604s) [noise=true , address=45.91.116.13:4154 , piece_id=X3EDJQ2O2BJJIXII57NHATPKJU64KIQXYSJGNSRSVKW2OPWVSA6Q, seg_pos={0 0}, upload=439635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 12:24:37.694252227 (500h55m37.565335272s) [noise=true , address=45.91.116.13:4154 , piece_id=X3EDJQ2O2BJJIXII57NHATPKJU64KIQXYSJGNSRSVKW2OPWVSA6Q, seg_pos={0 0}, upload=439635]: Long tail cancellation due to optimal threshold Dec 26 12:24:37.694416711 (500h55m37.565499756s) [noise=true , address=45.91.116.8:4046 , piece_id=XWUYYNVKZE2HMELUMRJGKRCTQ2N3N5GMCBQ7QCSIBS75IMRR5WBA, seg_pos={0 0}, upload=439635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 26 12:24:37.694426028 (500h55m37.565509063s) [noise=true , address=45.91.116.8:4046 , piece_id=XWUYYNVKZE2HMELUMRJGKRCTQ2N3N5GMCBQ7QCSIBS75IMRR5WBA, seg_pos={0 0}, upload=439635]: Long tail cancellation due to optimal threshold Dec 26 12:24:37.694592615 (500h55m37.565675691s) [noise=true , address=45.91.116.5:4299 , piece_id=XYGPHPG635SWQC4FK4BJ57CP7NS7FQ5SNYDIE6TXUZ4YEHXWOJ6Q, seg_pos={0 0}, upload=439635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 12:24:37.694703236 (500h55m37.565786291s) [noise=true , address=45.91.116.5:4299 , piece_id=XYGPHPG635SWQC4FK4BJ57CP7NS7FQ5SNYDIE6TXUZ4YEHXWOJ6Q, seg_pos={0 0}, upload=439635]: Long tail cancellation due to optimal threshold Dec 26 12:24:37.694728885 (500h55m37.56581192s) [seg_pos={0 0}, upload=439635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:24:37.694801714 (500h55m37.565884749s) [upload=439635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:24:37.694808055 (500h55m37.565891101s) [upload=439635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:24:37.983906490 (500h55m37.854989535s) [upload=439635]: Done uploading segments... Dec 26 12:24:37.983923403 (500h55m37.855006448s) [upload=439635]: Upload finished. err: <nil> Duration: 30.004284844s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [......X....X............XX........X..X......X.......XXX...X.] duration: 5.000551754s trace: https://jaeger.dev.storj.io/trace/77fa83e3c4f9fd80 FAIL (2025-12-26 12:24:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000574188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 42.084096197s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:24:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 42.084115815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X.X.X...X.X........X..XXXXX...X.X....X..X...X..XX.....XX] duration: 30.002958319s trace: https://jaeger.dev.storj.io/trace/7530410f887e4cbc FAIL (2025-12-26 12:24:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 12:23:48.541215431 (500h54m48.412298476s) [upload=439623]: Starting upload Dec 26 12:23:48.541297417 (500h54m48.412380462s) [upload=439623]: Done starting upload Dec 26 12:23:48.541373101 (500h54m48.412456136s) [upload=439623]: Uploading segments... Dec 26 12:23:48.541508459 (500h54m48.412591514s) [upload=439623]: Got next segment. Inline: false Dec 26 12:23:48.541550659 (500h54m48.412633694s) [upload=439623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:23:48.541555769 (500h54m48.412638804s) [upload=439623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:23:48.781434290 (500h54m48.652517325s) [seg_pos={0 0}, upload=439623]: Begin upload segment... Dec 26 12:23:48.781492901 (500h54m48.652575946s) [seg_pos={0 0}, upload=439623]: Done begin upload segment. Dec 26 12:23:48.781570620 (500h54m48.652653665s) [noise=true , address=45.91.116.11:4118 , piece_id=SDVVAPQGZJY2VANRNLDEEGWYCW2CUUGHBNHFPNQYBA6X3OOOWA5A, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.781605416 (500h54m48.652688451s) [noise=true , address=145.79.227.49:4338 , piece_id=IDTQS2BHMJJMVKGXHWD5DVUCQFLUIAS2T5ENV7RE6QKEU7DWAXHA, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.782186602 (500h54m48.653269637s) [noise=true , address=145.79.227.51:4411 , piece_id=GAVPNPW5QWPDBP6FY3JPWGZVEY7FIZYDX3FTSJVXNZO36PVJJWFQ, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.782596663 (500h54m48.653679708s) [noise=true , address=45.91.116.13:4155 , piece_id=4GRBMI4KUBI2JMESMD3ECHGEFDF7XWH22MK7RN2VLAG6ZLCX6VWQ, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.783115561 (500h54m48.654198606s) [noise=true , address=145.79.227.42:4121 , piece_id=BWMJQ45LXCAPHHGWE6FE3P5VOVTF4LZRJ45O43ZV3B5PK6IISHAA, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.783608520 (500h54m48.654691555s) [noise=true , address=145.79.227.11:4731 , piece_id=ENKIAHZOIN7TXUZ7CBUQ63CNBEXKC62BIB2EK432AYXQG5JEXYCA, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.784100516 (500h54m48.655183561s) [noise=true , address=45.91.116.9:4264 , piece_id=MOFRYTLED3OYUR4RWNG36PO36FJAW5QC3B2HJC2LYXSJFSL7PX6A, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.784700008 (500h54m48.655783043s) [noise=true , address=145.79.227.17:4970 , piece_id=GX224XN3EOW4PNHOC7ZRWRK4FUKV2QCBL6XS4TJON2SG4LXDMMCA, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.785246969 (500h54m48.656330005s) [noise=true , address=145.79.227.45:4230 , piece_id=ZIYKK4DR27ESMGAGAIQG3SYBXEVOG4TPGXF6VUCIKSQPHPM2IVOQ, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.785767040 (500h54m48.656850075s) [noise=true , address=145.79.227.44:4194 , piece_id=VBNQQVT4J4A3ZV4FTXUP7WWAWIZBXY76H2KTBSYSUNCBRTMVKA4Q, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.791442677 (500h54m48.662525722s) [noise=true , address=45.91.116.10:4082 , piece_id=FV6YN4IXQGFYJ5MJNY3YIOX6VYLEDALK7QOAWKYMWYW7EYD7ZOMQ, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.792319627 (500h54m48.663402662s) [noise=true , address=145.79.227.41:4085 , piece_id=DGVYUJN7GLMCZQZKLK7DBD6SPNEC5G6VVUJQDEIST5GTLSQYKKPA, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.796360622 (500h54m48.667443667s) [noise=true , address=145.79.227.48:4301 , piece_id=3DIPUHKTCQYND6WWJZC5OSZLDXMOLV74DM557DWVFN6T3X647ADQ, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.796737089 (500h54m48.667820124s) [noise=true , address=145.79.227.47:4267 , piece_id=BFFJTISHXO56RUFBJPR7SIZNBRQ2LOOJHOKHW2LUVBJK6HJBALEQ, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.798258155 (500h54m48.6693412s) [noise=true , address=145.79.227.16:4910 , piece_id=4AOSWJIJVQ4F7INJZGSTWHKQGA6DJOFWKQSHSSMEVYRBOZJBAY2A, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:48.799006900 (500h54m48.670089946s) [noise=true , address=145.79.227.50:4375 , piece_id=3C7G3BJNH7JF6TS32ZVSVSIAS753UMOVUQDMGLQ7D77MPZ262EQQ, seg_pos={0 0}, upload=439623]: Uploading piece... Dec 26 12:23:50.517289122 (500h54m50.388372157s) [upload=439623]: Next returned nil segment Dec 26 12:23:50.517310322 (500h54m50.388393377s) [upload=439623]: Waiting for error group managing segments... Dec 26 12:23:50.795905065 (500h54m50.6669881s) [noise=true , address=145.79.227.42:4121 , piece_id=BWMJQ45LXCAPHHGWE6FE3P5VOVTF4LZRJ45O43ZV3B5PK6IISHAA, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:50.798985911 (500h54m50.670068956s) [noise=true , address=145.79.227.17:4970 , piece_id=GX224XN3EOW4PNHOC7ZRWRK4FUKV2QCBL6XS4TJON2SG4LXDMMCA, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:50.801003042 (500h54m50.672086087s) [noise=true , address=145.79.227.41:4085 , piece_id=DGVYUJN7GLMCZQZKLK7DBD6SPNEC5G6VVUJQDEIST5GTLSQYKKPA, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:50.803121786 (500h54m50.674204821s) [noise=true , address=145.79.227.11:4731 , piece_id=ENKIAHZOIN7TXUZ7CBUQ63CNBEXKC62BIB2EK432AYXQG5JEXYCA, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:50.803691722 (500h54m50.674774787s) [noise=true , address=145.79.227.45:4230 , piece_id=ZIYKK4DR27ESMGAGAIQG3SYBXEVOG4TPGXF6VUCIKSQPHPM2IVOQ, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:50.803722139 (500h54m50.674805184s) [noise=true , address=145.79.227.48:4301 , piece_id=3DIPUHKTCQYND6WWJZC5OSZLDXMOLV74DM557DWVFN6T3X647ADQ, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:50.806078006 (500h54m50.677161041s) [noise=true , address=145.79.227.16:4910 , piece_id=4AOSWJIJVQ4F7INJZGSTWHKQGA6DJOFWKQSHSSMEVYRBOZJBAY2A, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:50.808907453 (500h54m50.679990488s) [noise=true , address=145.79.227.49:4338 , piece_id=IDTQS2BHMJJMVKGXHWD5DVUCQFLUIAS2T5ENV7RE6QKEU7DWAXHA, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:50.812701397 (500h54m50.683784443s) [noise=true , address=145.79.227.50:4375 , piece_id=3C7G3BJNH7JF6TS32ZVSVSIAS753UMOVUQDMGLQ7D77MPZ262EQQ, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:50.812929432 (500h54m50.684012487s) [noise=true , address=145.79.227.51:4411 , piece_id=GAVPNPW5QWPDBP6FY3JPWGZVEY7FIZYDX3FTSJVXNZO36PVJJWFQ, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:50.832719556 (500h54m50.703802602s) [noise=true , address=145.79.227.47:4267 , piece_id=BFFJTISHXO56RUFBJPR7SIZNBRQ2LOOJHOKHW2LUVBJK6HJBALEQ, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:51.061497299 (500h54m50.932580334s) [noise=true , address=45.91.116.10:4082 , piece_id=FV6YN4IXQGFYJ5MJNY3YIOX6VYLEDALK7QOAWKYMWYW7EYD7ZOMQ, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:51.116226432 (500h54m50.987309467s) [noise=true , address=45.91.116.11:4118 , piece_id=SDVVAPQGZJY2VANRNLDEEGWYCW2CUUGHBNHFPNQYBA6X3OOOWA5A, seg_pos={0 0}, upload=439623]: Done uploading piece. err: <nil> Dec 26 12:23:51.116247042 (500h54m50.987330077s) [seg_pos={0 0}, upload=439623]: Segment reached optimal threshold of 13 pieces. Dec 26 12:23:51.116467321 (500h54m50.987550366s) [noise=true , address=145.79.227.44:4194 , piece_id=VBNQQVT4J4A3ZV4FTXUP7WWAWIZBXY76H2KTBSYSUNCBRTMVKA4Q, seg_pos={0 0}, upload=439623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 12:23:51.116474264 (500h54m50.987557299s) [noise=true , address=145.79.227.44:4194 , piece_id=VBNQQVT4J4A3ZV4FTXUP7WWAWIZBXY76H2KTBSYSUNCBRTMVKA4Q, seg_pos={0 0}, upload=439623]: Long tail cancellation due to optimal threshold Dec 26 12:23:51.116692480 (500h54m50.987775535s) [noise=true , address=45.91.116.9:4264 , piece_id=MOFRYTLED3OYUR4RWNG36PO36FJAW5QC3B2HJC2LYXSJFSL7PX6A, seg_pos={0 0}, upload=439623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43566->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43566->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43566->45.91.116.9:4264: use of closed network connection Dec 26 12:23:51.116712848 (500h54m50.987795884s) [noise=true , address=45.91.116.9:4264 , piece_id=MOFRYTLED3OYUR4RWNG36PO36FJAW5QC3B2HJC2LYXSJFSL7PX6A, seg_pos={0 0}, upload=439623]: Long tail cancellation due to optimal threshold Dec 26 12:23:51.116749849 (500h54m50.987832914s) [noise=true , address=45.91.116.13:4155 , piece_id=4GRBMI4KUBI2JMESMD3ECHGEFDF7XWH22MK7RN2VLAG6ZLCX6VWQ, seg_pos={0 0}, upload=439623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 12:23:51.116761341 (500h54m50.987844376s) [noise=true , address=45.91.116.13:4155 , piece_id=4GRBMI4KUBI2JMESMD3ECHGEFDF7XWH22MK7RN2VLAG6ZLCX6VWQ, seg_pos={0 0}, upload=439623]: Long tail cancellation due to optimal threshold Dec 26 12:23:51.116787240 (500h54m50.987870275s) [seg_pos={0 0}, upload=439623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:23:51.116868596 (500h54m50.987951631s) [upload=439623]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:23:51.116876040 (500h54m50.987959085s) [upload=439623]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:23:51.397372453 (500h54m51.268455498s) [upload=439623]: Done uploading segments... Dec 26 12:23:51.397390508 (500h54m51.268473543s) [upload=439623]: Upload finished. err: <nil> Duration: 30.002971003s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.......X....X............XX........X..X......X.......XXX...X] duration: 5.001250055s trace: https://jaeger.dev.storj.io/trace/77fa83e3c4f9fd80 FAIL (2025-12-26 12:24:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001314077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 59.621594338s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:23:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 59.621611891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....X.X.X...X.X........X..XXXXX...X.X....X..X...X..XX.....X] duration: 30.002389624s trace: https://jaeger.dev.storj.io/trace/75f9316dc908d3ce FAIL (2025-12-26 12:23:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 12:23:18.330397582 (500h54m18.201480627s) [upload=439618]: Starting upload Dec 26 12:23:18.330489187 (500h54m18.201572232s) [upload=439618]: Done starting upload Dec 26 12:23:18.330706831 (500h54m18.201789876s) [upload=439618]: Uploading segments... Dec 26 12:23:18.330841428 (500h54m18.201924473s) [upload=439618]: Got next segment. Inline: false Dec 26 12:23:18.330902023 (500h54m18.201985058s) [upload=439618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:23:18.330909607 (500h54m18.201992642s) [upload=439618]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:23:18.572555444 (500h54m18.443638489s) [seg_pos={0 0}, upload=439618]: Begin upload segment... Dec 26 12:23:18.572618524 (500h54m18.443701569s) [seg_pos={0 0}, upload=439618]: Done begin upload segment. Dec 26 12:23:18.572688468 (500h54m18.443771503s) [noise=true , address=45.91.116.12:4226 , piece_id=63UUVBYW2PVVMNI2SCQ442SLC46RC653FF7DGLTYYW4ALDC275SQ, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.572710339 (500h54m18.443793374s) [noise=true , address=145.79.227.16:4909 , piece_id=CQZUZB3MBNXTU6MW4VKBTBAKRLQD6BVJT5UVGT4BQ3DQAAOZ4JIQ, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.573199209 (500h54m18.444282245s) [noise=true , address=45.91.116.9:4263 , piece_id=K5INLQBXIC3NQLZXGDYQ66D3VGP3Y4IZUEEX72VN3GTVJZXCNJZQ, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.573673393 (500h54m18.444756438s) [noise=true , address=145.79.227.31:4799 , piece_id=GMLRIF6QZGEVILKUC6LLYYYB232YCLIS4SB3D7FDVDR6IZFCSJPA, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.574213481 (500h54m18.445296526s) [noise=true , address=77.90.31.19:4842 , piece_id=CN5YSEFQXIOPB7ARN32QS5JOGUKUDNXG7YKXELXA2JJFDUIG2NWA, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.574579017 (500h54m18.445662052s) [noise=true , address=145.79.227.35:4483 , piece_id=BJ53JDWHL7XEPG5RVPLYZYSY6UFF3CUHFVX5WYFPRK3ZJLWRBTPA, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.575001922 (500h54m18.446084967s) [noise=true , address=145.79.227.48:4303 , piece_id=R6NNAH7NTDMTX4WUBOAD6DEL7IRVTD2YXDVGCOKJVXZ4UPKZYJIA, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.576553377 (500h54m18.447636422s) [noise=true , address=77.90.31.14:4661 , piece_id=T4E7SPXXFJ5XNQKHYIIDYRVAEJYE3ERTGAF6WSE4P44VSZ4YZWWA, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.577970334 (500h54m18.44905338s) [noise=true , address=145.79.227.4:4185 , piece_id=EDI4C2YZ4RXECRCI57DCDT6YBJ2GU75OZVGYRTIRK2JGPOXZFTZQ, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.579669500 (500h54m18.450752545s) [noise=true , address=145.79.227.45:4229 , piece_id=6ZRXICOKQJ2X32XFBD7DLAPARP2M6YZNF52Q2RDYYHVVJWM76AJQ, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.581413591 (500h54m18.452496636s) [noise=true , address=145.79.227.19:4078 , piece_id=HBCKZLQ4VXUPLHN5SCIA5P7EAWYBKK7JM2OJ7RUGAYM4SJQCTFTA, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.581494215 (500h54m18.45257726s) [noise=true , address=145.79.227.33:4871 , piece_id=RRWDIZQQ45OWAUTUBVVXLNDDAGLC3ZMVI37622MW4MSTVBTKPTXA, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.582058299 (500h54m18.453141334s) [noise=true , address=145.79.227.36:4518 , piece_id=WQALZFIJGXLGBXJUSY6GLILXO7QRBPM7O4M7VVEGNJFJA26BKV4A, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.586051914 (500h54m18.457134949s) [noise=true , address=77.90.31.11:4556 , piece_id=GFYX2EWP5ZPQPUNUYYYQZRBCRJIXLVUSQQUZQRVNW7AEZ52NXGLQ, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.590369846 (500h54m18.461452881s) [noise=true , address=145.79.227.29:4941 , piece_id=4CVKJQAY4MXTP5HNPK7PTMUJJ3LKRJ434XNM2YGFDCIYSF6L5Y2A, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:18.593203491 (500h54m18.464286536s) [noise=true , address=45.91.116.4:4940 , piece_id=OWVKQEQ2I7B2UV45LLFVN5RN6KOTJP7QLSQQSMYVMBKT5MIWBDJA, seg_pos={0 0}, upload=439618]: Uploading piece... Dec 26 12:23:20.290311972 (500h54m20.161395008s) [upload=439618]: Next returned nil segment Dec 26 12:23:20.290329426 (500h54m20.161412471s) [upload=439618]: Waiting for error group managing segments... Dec 26 12:23:20.580835450 (500h54m20.451918495s) [noise=true , address=145.79.227.19:4078 , piece_id=HBCKZLQ4VXUPLHN5SCIA5P7EAWYBKK7JM2OJ7RUGAYM4SJQCTFTA, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:20.580920011 (500h54m20.452003056s) [noise=true , address=145.79.227.16:4909 , piece_id=CQZUZB3MBNXTU6MW4VKBTBAKRLQD6BVJT5UVGT4BQ3DQAAOZ4JIQ, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:20.581448958 (500h54m20.452532003s) [noise=true , address=145.79.227.48:4303 , piece_id=R6NNAH7NTDMTX4WUBOAD6DEL7IRVTD2YXDVGCOKJVXZ4UPKZYJIA, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:20.582057558 (500h54m20.453140603s) [noise=true , address=145.79.227.35:4483 , piece_id=BJ53JDWHL7XEPG5RVPLYZYSY6UFF3CUHFVX5WYFPRK3ZJLWRBTPA, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:20.585724219 (500h54m20.456807264s) [noise=true , address=145.79.227.31:4799 , piece_id=GMLRIF6QZGEVILKUC6LLYYYB232YCLIS4SB3D7FDVDR6IZFCSJPA, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:20.586682303 (500h54m20.457765348s) [noise=true , address=145.79.227.29:4941 , piece_id=4CVKJQAY4MXTP5HNPK7PTMUJJ3LKRJ434XNM2YGFDCIYSF6L5Y2A, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:20.604528986 (500h54m20.475612022s) [noise=true , address=145.79.227.36:4518 , piece_id=WQALZFIJGXLGBXJUSY6GLILXO7QRBPM7O4M7VVEGNJFJA26BKV4A, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:20.612455220 (500h54m20.483538255s) [noise=true , address=145.79.227.45:4229 , piece_id=6ZRXICOKQJ2X32XFBD7DLAPARP2M6YZNF52Q2RDYYHVVJWM76AJQ, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:20.731949197 (500h54m20.603032242s) [noise=true , address=145.79.227.33:4871 , piece_id=RRWDIZQQ45OWAUTUBVVXLNDDAGLC3ZMVI37622MW4MSTVBTKPTXA, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:20.924275069 (500h54m20.795358104s) [noise=true , address=77.90.31.14:4661 , piece_id=T4E7SPXXFJ5XNQKHYIIDYRVAEJYE3ERTGAF6WSE4P44VSZ4YZWWA, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:21.483169983 (500h54m21.354253018s) [noise=true , address=77.90.31.19:4842 , piece_id=CN5YSEFQXIOPB7ARN32QS5JOGUKUDNXG7YKXELXA2JJFDUIG2NWA, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:21.569452380 (500h54m21.440535425s) [noise=true , address=45.91.116.4:4940 , piece_id=OWVKQEQ2I7B2UV45LLFVN5RN6KOTJP7QLSQQSMYVMBKT5MIWBDJA, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:21.919875537 (500h54m21.790958582s) [noise=true , address=45.91.116.9:4263 , piece_id=K5INLQBXIC3NQLZXGDYQ66D3VGP3Y4IZUEEX72VN3GTVJZXCNJZQ, seg_pos={0 0}, upload=439618]: Done uploading piece. err: <nil> Dec 26 12:23:21.919897107 (500h54m21.790980142s) [seg_pos={0 0}, upload=439618]: Segment reached optimal threshold of 13 pieces. Dec 26 12:23:21.920094794 (500h54m21.791177839s) [noise=true , address=145.79.227.4:4185 , piece_id=EDI4C2YZ4RXECRCI57DCDT6YBJ2GU75OZVGYRTIRK2JGPOXZFTZQ, seg_pos={0 0}, upload=439618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:44006->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:44006->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44006->145.79.227.4:4185: use of closed network connection Dec 26 12:23:21.920103320 (500h54m21.791186365s) [noise=true , address=145.79.227.4:4185 , piece_id=EDI4C2YZ4RXECRCI57DCDT6YBJ2GU75OZVGYRTIRK2JGPOXZFTZQ, seg_pos={0 0}, upload=439618]: Long tail cancellation due to optimal threshold Dec 26 12:23:21.920143577 (500h54m21.791226622s) [noise=true , address=77.90.31.11:4556 , piece_id=GFYX2EWP5ZPQPUNUYYYQZRBCRJIXLVUSQQUZQRVNW7AEZ52NXGLQ, seg_pos={0 0}, upload=439618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 26 12:23:21.920152164 (500h54m21.791235199s) [noise=true , address=77.90.31.11:4556 , piece_id=GFYX2EWP5ZPQPUNUYYYQZRBCRJIXLVUSQQUZQRVNW7AEZ52NXGLQ, seg_pos={0 0}, upload=439618]: Long tail cancellation due to optimal threshold Dec 26 12:23:21.920367614 (500h54m21.791450659s) [noise=true , address=45.91.116.12:4226 , piece_id=63UUVBYW2PVVMNI2SCQ442SLC46RC653FF7DGLTYYW4ALDC275SQ, seg_pos={0 0}, upload=439618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 12:23:21.920375949 (500h54m21.791458994s) [noise=true , address=45.91.116.12:4226 , piece_id=63UUVBYW2PVVMNI2SCQ442SLC46RC653FF7DGLTYYW4ALDC275SQ, seg_pos={0 0}, upload=439618]: Long tail cancellation due to optimal threshold Dec 26 12:23:21.920395006 (500h54m21.791478041s) [seg_pos={0 0}, upload=439618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:23:21.920489525 (500h54m21.79157256s) [upload=439618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:23:21.920495217 (500h54m21.791578262s) [upload=439618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:23:22.211715561 (500h54m22.082798626s) [upload=439618]: Done uploading segments... Dec 26 12:23:22.211740078 (500h54m22.082823113s) [upload=439618]: Upload finished. err: <nil> Duration: 30.00241385s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [........X....X............XX........X..X......X.......XXX...] duration: 4.122617474s trace: https://jaeger.dev.storj.io/trace/44ed282b049beec8 FAIL (2025-12-26 12:23:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.122644896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 51.651316806s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:22:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 51.651331825s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......X.X.X...X.X........X..XXXXX...X.X....X..X...X..XX.....] duration: 5.691505712s trace: https://jaeger.dev.storj.io/trace/1ca2c6e448115a69 FAIL (2025-12-26 12:22:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ) context deadline exceeded ======LOGS====== Dec 26 12:19:41.876746940 (500h50m41.747829985s) [upload=439567]: Starting upload Dec 26 12:19:41.876800212 (500h50m41.747883257s) [upload=439567]: Done starting upload Dec 26 12:19:41.876896925 (500h50m41.747979971s) [upload=439567]: Uploading segments... Dec 26 12:19:41.877029057 (500h50m41.748112102s) [upload=439567]: Got next segment. Inline: false Dec 26 12:19:41.877072199 (500h50m41.748155234s) [upload=439567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:19:41.877275967 (500h50m41.748359002s) [upload=439567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:19:42.123766398 (500h50m41.994849433s) [seg_pos={0 0}, upload=439567]: Begin upload segment... Dec 26 12:19:42.123844566 (500h50m41.994927601s) [seg_pos={0 0}, upload=439567]: Done begin upload segment. Dec 26 12:19:42.123902998 (500h50m41.994986033s) [noise=true , address=145.79.227.37:4565 , piece_id=UQEMY2CU46IWOPC5FJ6ZQ3L2SVFFMN2AMHWFVLC666ASBX5HZP6A, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.123935851 (500h50m41.995018886s) [noise=true , address=77.90.31.17:4773 , piece_id=7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.124479416 (500h50m41.995562461s) [noise=true , address=45.91.116.9:4261 , piece_id=CX3XDNAFOUK5CY2Y3EE2APMTQWLGGCEVQNMR7LR2CZMOD4RC3NYQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.124732478 (500h50m41.995815513s) [noise=true , address=145.79.227.36:4518 , piece_id=YGFS6XUJ5INA7VW4DQKHPA73T5VUSQPPKNO4GQUXRCDWLDTBTC7Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125057166 (500h50m41.996140201s) [noise=true , address=145.79.227.19:4078 , piece_id=MBW6DTEEMKS3JUFHY7TV5BRTYCC6MRXIB2NY2E7YLGMIPPGSDYNA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125359622 (500h50m41.996442667s) [noise=true , address=145.79.227.2:4113 , piece_id=JA3KVITDVP52CHNLGH4C6VCJQOQQFQM4J3BMWKF7OAAJWIWPBURA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125398275 (500h50m41.99648131s) [noise=true , address=145.79.227.39:4014 , piece_id=K6LK5ZSL3EUODWK7PJCDYLRDMVLFMAV7557KL6QZOGYHBNXE626Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125649193 (500h50m41.996732228s) [noise=true , address=145.79.227.5:4223 , piece_id=KDRA5HRL35MD7JYJOVTO4M66CD3PM5OLWTKF7MIQKR2YY2ZTVIOA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125827873 (500h50m41.996910918s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125953412 (500h50m41.997036457s) [noise=true , address=145.79.227.14:4840 , piece_id=OEVFYAAICKFEBD6MAVYHGLFQOZJPVZPITAOC3PJCLCHJTJ5UIXHQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126228617 (500h50m41.997311652s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126240208 (500h50m41.997323253s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126533167 (500h50m41.997616202s) [noise=true , address=45.91.116.7:4010 , piece_id=Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126594142 (500h50m41.997677177s) [noise=true , address=145.79.227.45:4231 , piece_id=IP2NOXBJ2KEYYAP3JR3XXYSQP5YDR36ILBHI6Y53L2QL7U5DLGQA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.127000256 (500h50m41.998083301s) [noise=true , address=145.79.227.4:4187 , piece_id=UWJPDDXWWGZUZYOGJ4I6NXI56GSFRI2KJWMZYJQA2HIDC7NG76XA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.128148763 (500h50m41.999231808s) [noise=true , address=45.91.116.5:4297 , piece_id=NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:43.831946619 (500h50m43.703029654s) [upload=439567]: Next returned nil segment Dec 26 12:19:43.831960697 (500h50m43.703043732s) [upload=439567]: Waiting for error group managing segments... Dec 26 12:19:44.127538768 (500h50m43.998621813s) [noise=true , address=145.79.227.5:4223 , piece_id=KDRA5HRL35MD7JYJOVTO4M66CD3PM5OLWTKF7MIQKR2YY2ZTVIOA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.129415963 (500h50m44.000498998s) [noise=true , address=145.79.227.45:4231 , piece_id=IP2NOXBJ2KEYYAP3JR3XXYSQP5YDR36ILBHI6Y53L2QL7U5DLGQA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.131062498 (500h50m44.002145533s) [noise=true , address=145.79.227.2:4113 , piece_id=JA3KVITDVP52CHNLGH4C6VCJQOQQFQM4J3BMWKF7OAAJWIWPBURA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136261638 (500h50m44.007344673s) [noise=true , address=145.79.227.39:4014 , piece_id=K6LK5ZSL3EUODWK7PJCDYLRDMVLFMAV7557KL6QZOGYHBNXE626Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136307365 (500h50m44.00739041s) [noise=true , address=145.79.227.36:4518 , piece_id=YGFS6XUJ5INA7VW4DQKHPA73T5VUSQPPKNO4GQUXRCDWLDTBTC7Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136783271 (500h50m44.007866306s) [noise=true , address=145.79.227.4:4187 , piece_id=UWJPDDXWWGZUZYOGJ4I6NXI56GSFRI2KJWMZYJQA2HIDC7NG76XA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.137438258 (500h50m44.008521293s) [noise=true , address=145.79.227.14:4840 , piece_id=OEVFYAAICKFEBD6MAVYHGLFQOZJPVZPITAOC3PJCLCHJTJ5UIXHQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.151198460 (500h50m44.022281505s) [noise=true , address=145.79.227.19:4078 , piece_id=MBW6DTEEMKS3JUFHY7TV5BRTYCC6MRXIB2NY2E7YLGMIPPGSDYNA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.160248853 (500h50m44.031331888s) [noise=true , address=145.79.227.37:4565 , piece_id=UQEMY2CU46IWOPC5FJ6ZQ3L2SVFFMN2AMHWFVLC666ASBX5HZP6A, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.461365753 (500h50m44.332448798s) [noise=true , address=45.91.116.7:4010 , piece_id=Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.500371746 (500h50m44.371454792s) [noise=true , address=45.91.116.5:4297 , piece_id=NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.523280448 (500h50m44.394363484s) [noise=true , address=77.90.31.17:4773 , piece_id=7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.662224932 (500h50m44.533307977s) [noise=true , address=45.91.116.9:4261 , piece_id=CX3XDNAFOUK5CY2Y3EE2APMTQWLGGCEVQNMR7LR2CZMOD4RC3NYQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.662279276 (500h50m44.533362311s) [seg_pos={0 0}, upload=439567]: Segment reached optimal threshold of 13 pieces. Dec 26 12:19:44.662454219 (500h50m44.533537264s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection Dec 26 12:19:44.662462445 (500h50m44.53354548s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662489166 (500h50m44.533572211s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:44.662497331 (500h50m44.533580366s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662782124 (500h50m44.533865169s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: context canceled; context canceled Dec 26 12:19:44.662792744 (500h50m44.533875789s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662812070 (500h50m44.533895116s) [seg_pos={0 0}, upload=439567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:19:44.662945435 (500h50m44.53402848s) [upload=439567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:44.662951746 (500h50m44.534034792s) [upload=439567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:44.972002824 (500h50m44.843085859s) [upload=439567]: Done uploading segments... Dec 26 12:19:44.972021861 (500h50m44.843104906s) [upload=439567]: Upload finished. err: <nil> Duration: 5.691523976s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 33.425537919s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:22:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 33.425557255s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.........X....X............XX........X..X......X.......XXX..] duration: 3.452853899s trace: https://jaeger.dev.storj.io/trace/44ed282b049beec8 FAIL (2025-12-26 12:21:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.452892914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X......X.X.X...X.X........X..XXXXX...X.X....X..X...X..XX....] duration: 6.241695159s trace: https://jaeger.dev.storj.io/trace/1ca2c6e448115a69 FAIL (2025-12-26 12:21:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ) context deadline exceeded ======LOGS====== Dec 26 12:19:41.876746940 (500h50m41.747829985s) [upload=439567]: Starting upload Dec 26 12:19:41.876800212 (500h50m41.747883257s) [upload=439567]: Done starting upload Dec 26 12:19:41.876896925 (500h50m41.747979971s) [upload=439567]: Uploading segments... Dec 26 12:19:41.877029057 (500h50m41.748112102s) [upload=439567]: Got next segment. Inline: false Dec 26 12:19:41.877072199 (500h50m41.748155234s) [upload=439567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:19:41.877275967 (500h50m41.748359002s) [upload=439567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:19:42.123766398 (500h50m41.994849433s) [seg_pos={0 0}, upload=439567]: Begin upload segment... Dec 26 12:19:42.123844566 (500h50m41.994927601s) [seg_pos={0 0}, upload=439567]: Done begin upload segment. Dec 26 12:19:42.123902998 (500h50m41.994986033s) [noise=true , address=145.79.227.37:4565 , piece_id=UQEMY2CU46IWOPC5FJ6ZQ3L2SVFFMN2AMHWFVLC666ASBX5HZP6A, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.123935851 (500h50m41.995018886s) [noise=true , address=77.90.31.17:4773 , piece_id=7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.124479416 (500h50m41.995562461s) [noise=true , address=45.91.116.9:4261 , piece_id=CX3XDNAFOUK5CY2Y3EE2APMTQWLGGCEVQNMR7LR2CZMOD4RC3NYQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.124732478 (500h50m41.995815513s) [noise=true , address=145.79.227.36:4518 , piece_id=YGFS6XUJ5INA7VW4DQKHPA73T5VUSQPPKNO4GQUXRCDWLDTBTC7Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125057166 (500h50m41.996140201s) [noise=true , address=145.79.227.19:4078 , piece_id=MBW6DTEEMKS3JUFHY7TV5BRTYCC6MRXIB2NY2E7YLGMIPPGSDYNA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125359622 (500h50m41.996442667s) [noise=true , address=145.79.227.2:4113 , piece_id=JA3KVITDVP52CHNLGH4C6VCJQOQQFQM4J3BMWKF7OAAJWIWPBURA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125398275 (500h50m41.99648131s) [noise=true , address=145.79.227.39:4014 , piece_id=K6LK5ZSL3EUODWK7PJCDYLRDMVLFMAV7557KL6QZOGYHBNXE626Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125649193 (500h50m41.996732228s) [noise=true , address=145.79.227.5:4223 , piece_id=KDRA5HRL35MD7JYJOVTO4M66CD3PM5OLWTKF7MIQKR2YY2ZTVIOA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125827873 (500h50m41.996910918s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125953412 (500h50m41.997036457s) [noise=true , address=145.79.227.14:4840 , piece_id=OEVFYAAICKFEBD6MAVYHGLFQOZJPVZPITAOC3PJCLCHJTJ5UIXHQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126228617 (500h50m41.997311652s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126240208 (500h50m41.997323253s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126533167 (500h50m41.997616202s) [noise=true , address=45.91.116.7:4010 , piece_id=Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126594142 (500h50m41.997677177s) [noise=true , address=145.79.227.45:4231 , piece_id=IP2NOXBJ2KEYYAP3JR3XXYSQP5YDR36ILBHI6Y53L2QL7U5DLGQA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.127000256 (500h50m41.998083301s) [noise=true , address=145.79.227.4:4187 , piece_id=UWJPDDXWWGZUZYOGJ4I6NXI56GSFRI2KJWMZYJQA2HIDC7NG76XA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.128148763 (500h50m41.999231808s) [noise=true , address=45.91.116.5:4297 , piece_id=NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:43.831946619 (500h50m43.703029654s) [upload=439567]: Next returned nil segment Dec 26 12:19:43.831960697 (500h50m43.703043732s) [upload=439567]: Waiting for error group managing segments... Dec 26 12:19:44.127538768 (500h50m43.998621813s) [noise=true , address=145.79.227.5:4223 , piece_id=KDRA5HRL35MD7JYJOVTO4M66CD3PM5OLWTKF7MIQKR2YY2ZTVIOA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.129415963 (500h50m44.000498998s) [noise=true , address=145.79.227.45:4231 , piece_id=IP2NOXBJ2KEYYAP3JR3XXYSQP5YDR36ILBHI6Y53L2QL7U5DLGQA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.131062498 (500h50m44.002145533s) [noise=true , address=145.79.227.2:4113 , piece_id=JA3KVITDVP52CHNLGH4C6VCJQOQQFQM4J3BMWKF7OAAJWIWPBURA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136261638 (500h50m44.007344673s) [noise=true , address=145.79.227.39:4014 , piece_id=K6LK5ZSL3EUODWK7PJCDYLRDMVLFMAV7557KL6QZOGYHBNXE626Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136307365 (500h50m44.00739041s) [noise=true , address=145.79.227.36:4518 , piece_id=YGFS6XUJ5INA7VW4DQKHPA73T5VUSQPPKNO4GQUXRCDWLDTBTC7Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136783271 (500h50m44.007866306s) [noise=true , address=145.79.227.4:4187 , piece_id=UWJPDDXWWGZUZYOGJ4I6NXI56GSFRI2KJWMZYJQA2HIDC7NG76XA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.137438258 (500h50m44.008521293s) [noise=true , address=145.79.227.14:4840 , piece_id=OEVFYAAICKFEBD6MAVYHGLFQOZJPVZPITAOC3PJCLCHJTJ5UIXHQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.151198460 (500h50m44.022281505s) [noise=true , address=145.79.227.19:4078 , piece_id=MBW6DTEEMKS3JUFHY7TV5BRTYCC6MRXIB2NY2E7YLGMIPPGSDYNA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.160248853 (500h50m44.031331888s) [noise=true , address=145.79.227.37:4565 , piece_id=UQEMY2CU46IWOPC5FJ6ZQ3L2SVFFMN2AMHWFVLC666ASBX5HZP6A, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.461365753 (500h50m44.332448798s) [noise=true , address=45.91.116.7:4010 , piece_id=Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.500371746 (500h50m44.371454792s) [noise=true , address=45.91.116.5:4297 , piece_id=NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.523280448 (500h50m44.394363484s) [noise=true , address=77.90.31.17:4773 , piece_id=7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.662224932 (500h50m44.533307977s) [noise=true , address=45.91.116.9:4261 , piece_id=CX3XDNAFOUK5CY2Y3EE2APMTQWLGGCEVQNMR7LR2CZMOD4RC3NYQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.662279276 (500h50m44.533362311s) [seg_pos={0 0}, upload=439567]: Segment reached optimal threshold of 13 pieces. Dec 26 12:19:44.662454219 (500h50m44.533537264s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection Dec 26 12:19:44.662462445 (500h50m44.53354548s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662489166 (500h50m44.533572211s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:44.662497331 (500h50m44.533580366s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662782124 (500h50m44.533865169s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: context canceled; context canceled Dec 26 12:19:44.662792744 (500h50m44.533875789s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662812070 (500h50m44.533895116s) [seg_pos={0 0}, upload=439567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:19:44.662945435 (500h50m44.53402848s) [upload=439567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:44.662951746 (500h50m44.534034792s) [upload=439567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:44.972002824 (500h50m44.843085859s) [upload=439567]: Done uploading segments... Dec 26 12:19:44.972021861 (500h50m44.843104906s) [upload=439567]: Upload finished. err: <nil> Duration: 6.241721538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 47.949954774s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:21:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 47.949973048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X......X.X.X...X.X........X..XXXXX...X.X....X..X...X..XX...] duration: 6.534873671s trace: https://jaeger.dev.storj.io/trace/1ca2c6e448115a69 FAIL (2025-12-26 12:20:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ) context deadline exceeded ======LOGS====== Dec 26 12:19:41.876746940 (500h50m41.747829985s) [upload=439567]: Starting upload Dec 26 12:19:41.876800212 (500h50m41.747883257s) [upload=439567]: Done starting upload Dec 26 12:19:41.876896925 (500h50m41.747979971s) [upload=439567]: Uploading segments... Dec 26 12:19:41.877029057 (500h50m41.748112102s) [upload=439567]: Got next segment. Inline: false Dec 26 12:19:41.877072199 (500h50m41.748155234s) [upload=439567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:19:41.877275967 (500h50m41.748359002s) [upload=439567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:19:42.123766398 (500h50m41.994849433s) [seg_pos={0 0}, upload=439567]: Begin upload segment... Dec 26 12:19:42.123844566 (500h50m41.994927601s) [seg_pos={0 0}, upload=439567]: Done begin upload segment. Dec 26 12:19:42.123902998 (500h50m41.994986033s) [noise=true , address=145.79.227.37:4565 , piece_id=UQEMY2CU46IWOPC5FJ6ZQ3L2SVFFMN2AMHWFVLC666ASBX5HZP6A, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.123935851 (500h50m41.995018886s) [noise=true , address=77.90.31.17:4773 , piece_id=7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.124479416 (500h50m41.995562461s) [noise=true , address=45.91.116.9:4261 , piece_id=CX3XDNAFOUK5CY2Y3EE2APMTQWLGGCEVQNMR7LR2CZMOD4RC3NYQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.124732478 (500h50m41.995815513s) [noise=true , address=145.79.227.36:4518 , piece_id=YGFS6XUJ5INA7VW4DQKHPA73T5VUSQPPKNO4GQUXRCDWLDTBTC7Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125057166 (500h50m41.996140201s) [noise=true , address=145.79.227.19:4078 , piece_id=MBW6DTEEMKS3JUFHY7TV5BRTYCC6MRXIB2NY2E7YLGMIPPGSDYNA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125359622 (500h50m41.996442667s) [noise=true , address=145.79.227.2:4113 , piece_id=JA3KVITDVP52CHNLGH4C6VCJQOQQFQM4J3BMWKF7OAAJWIWPBURA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125398275 (500h50m41.99648131s) [noise=true , address=145.79.227.39:4014 , piece_id=K6LK5ZSL3EUODWK7PJCDYLRDMVLFMAV7557KL6QZOGYHBNXE626Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125649193 (500h50m41.996732228s) [noise=true , address=145.79.227.5:4223 , piece_id=KDRA5HRL35MD7JYJOVTO4M66CD3PM5OLWTKF7MIQKR2YY2ZTVIOA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125827873 (500h50m41.996910918s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125953412 (500h50m41.997036457s) [noise=true , address=145.79.227.14:4840 , piece_id=OEVFYAAICKFEBD6MAVYHGLFQOZJPVZPITAOC3PJCLCHJTJ5UIXHQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126228617 (500h50m41.997311652s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126240208 (500h50m41.997323253s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126533167 (500h50m41.997616202s) [noise=true , address=45.91.116.7:4010 , piece_id=Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126594142 (500h50m41.997677177s) [noise=true , address=145.79.227.45:4231 , piece_id=IP2NOXBJ2KEYYAP3JR3XXYSQP5YDR36ILBHI6Y53L2QL7U5DLGQA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.127000256 (500h50m41.998083301s) [noise=true , address=145.79.227.4:4187 , piece_id=UWJPDDXWWGZUZYOGJ4I6NXI56GSFRI2KJWMZYJQA2HIDC7NG76XA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.128148763 (500h50m41.999231808s) [noise=true , address=45.91.116.5:4297 , piece_id=NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:43.831946619 (500h50m43.703029654s) [upload=439567]: Next returned nil segment Dec 26 12:19:43.831960697 (500h50m43.703043732s) [upload=439567]: Waiting for error group managing segments... Dec 26 12:19:44.127538768 (500h50m43.998621813s) [noise=true , address=145.79.227.5:4223 , piece_id=KDRA5HRL35MD7JYJOVTO4M66CD3PM5OLWTKF7MIQKR2YY2ZTVIOA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.129415963 (500h50m44.000498998s) [noise=true , address=145.79.227.45:4231 , piece_id=IP2NOXBJ2KEYYAP3JR3XXYSQP5YDR36ILBHI6Y53L2QL7U5DLGQA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.131062498 (500h50m44.002145533s) [noise=true , address=145.79.227.2:4113 , piece_id=JA3KVITDVP52CHNLGH4C6VCJQOQQFQM4J3BMWKF7OAAJWIWPBURA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136261638 (500h50m44.007344673s) [noise=true , address=145.79.227.39:4014 , piece_id=K6LK5ZSL3EUODWK7PJCDYLRDMVLFMAV7557KL6QZOGYHBNXE626Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136307365 (500h50m44.00739041s) [noise=true , address=145.79.227.36:4518 , piece_id=YGFS6XUJ5INA7VW4DQKHPA73T5VUSQPPKNO4GQUXRCDWLDTBTC7Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136783271 (500h50m44.007866306s) [noise=true , address=145.79.227.4:4187 , piece_id=UWJPDDXWWGZUZYOGJ4I6NXI56GSFRI2KJWMZYJQA2HIDC7NG76XA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.137438258 (500h50m44.008521293s) [noise=true , address=145.79.227.14:4840 , piece_id=OEVFYAAICKFEBD6MAVYHGLFQOZJPVZPITAOC3PJCLCHJTJ5UIXHQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.151198460 (500h50m44.022281505s) [noise=true , address=145.79.227.19:4078 , piece_id=MBW6DTEEMKS3JUFHY7TV5BRTYCC6MRXIB2NY2E7YLGMIPPGSDYNA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.160248853 (500h50m44.031331888s) [noise=true , address=145.79.227.37:4565 , piece_id=UQEMY2CU46IWOPC5FJ6ZQ3L2SVFFMN2AMHWFVLC666ASBX5HZP6A, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.461365753 (500h50m44.332448798s) [noise=true , address=45.91.116.7:4010 , piece_id=Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.500371746 (500h50m44.371454792s) [noise=true , address=45.91.116.5:4297 , piece_id=NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.523280448 (500h50m44.394363484s) [noise=true , address=77.90.31.17:4773 , piece_id=7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.662224932 (500h50m44.533307977s) [noise=true , address=45.91.116.9:4261 , piece_id=CX3XDNAFOUK5CY2Y3EE2APMTQWLGGCEVQNMR7LR2CZMOD4RC3NYQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.662279276 (500h50m44.533362311s) [seg_pos={0 0}, upload=439567]: Segment reached optimal threshold of 13 pieces. Dec 26 12:19:44.662454219 (500h50m44.533537264s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection Dec 26 12:19:44.662462445 (500h50m44.53354548s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662489166 (500h50m44.533572211s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:44.662497331 (500h50m44.533580366s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662782124 (500h50m44.533865169s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: context canceled; context canceled Dec 26 12:19:44.662792744 (500h50m44.533875789s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662812070 (500h50m44.533895116s) [seg_pos={0 0}, upload=439567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:19:44.662945435 (500h50m44.53402848s) [upload=439567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:44.662951746 (500h50m44.534034792s) [upload=439567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:44.972002824 (500h50m44.843085859s) [upload=439567]: Done uploading segments... Dec 26 12:19:44.972021861 (500h50m44.843104906s) [upload=439567]: Upload finished. err: <nil> Duration: 6.534899711s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..........X....X............XX........X..X......X.......XXX.] duration: 3.388486264s trace: https://jaeger.dev.storj.io/trace/44ed282b049beec8 FAIL (2025-12-26 12:20:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.388518005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.003401299s trace: https://jaeger.dev.storj.io/trace/6d3b95e327be2c54 FAIL (2025-12-26 12:20:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.679750639 (500h50m39.550833684s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680136643 (500h50m39.551219689s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680492952 (500h50m39.551575987s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.680922460 (500h50m39.552005505s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681276274 (500h50m39.552359319s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.681654093 (500h50m39.552737138s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682067039 (500h50m39.553150084s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.682379774 (500h50m39.553462819s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683041544 (500h50m39.554124589s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683570462 (500h50m39.554653507s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.683705018 (500h50m39.554788053s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684085282 (500h50m39.555168327s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684406755 (500h50m39.55548979s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.684706585 (500h50m39.55578962s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:39.685024661 (500h50m39.556107706s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Uploading piece... Dec 26 12:19:41.035952792 (500h50m40.907035837s) [upload=439566]: Next returned nil segment Dec 26 12:19:41.035976698 (500h50m40.907059733s) [upload=439566]: Waiting for error group managing segments... Dec 26 12:19:41.377687444 (500h50m41.248770489s) [noise=true , address=64.49.13.154:4931 , piece_id=UJ4EPADH5M5WOTG4GYRXPVCQTGOJJ4NFBAROP7LYVXFHAXDUVUZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.548443033 (500h50m41.419526068s) [noise=true , address=145.79.227.52:4460 , piece_id=YOIVEJINACCT5INWUJHUO2XWXJ36NHJB7QWCHECUEKFRNLE5IQNA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.599484394 (500h50m41.470567439s) [noise=true , address=145.79.227.49:4340 , piece_id=T4JHPE7ZXBFG5XTGEHOEWISOSQV6ZIUGG2UM73LJBJAFBFRNSROQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.716021842 (500h50m41.587104887s) [noise=true , address=77.90.31.19:4872 , piece_id=PXLKRVVD7HRRQHY5JMA5PJ5E2DLVCTMEPEBEPKFIDYE5XTNAZCJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.754929699 (500h50m41.626012735s) [noise=true , address=77.90.31.13:4656 , piece_id=D6LR6ZQR5OJJYYYQBVSY3DDV7X42EMJS34BHWW3UFXVURQPVWWGA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.762853698 (500h50m41.633936743s) [noise=true , address=45.91.116.7:4015 , piece_id=A6AIUI5HCWAEBJWPKGAWHKG4N5MQYMRHPPR7DFEMWXPNRRP5DPWQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.785565565 (500h50m41.6566486s) [noise=true , address=64.49.13.170:4718 , piece_id=ZUWQHTJNYO3JAXM3QMLR7Q7C5O2ZZ5NUYJJB52NV7IFVWT6LXGUQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.786490536 (500h50m41.657573571s) [noise=true , address=45.91.116.12:4238 , piece_id=OKYV6LGXDXPFN33O476PSHA6OHKWMVPQ2FPLF4ERN3E7FG4CDOVQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:41.809463170 (500h50m41.680546205s) [noise=true , address=45.91.116.13:4176 , piece_id=FAHNEYIZBDM6ZITHRCD6I7ZVNZ22L2TAO5VDRND3RT7MG5AYFKRQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.498537797 (500h50m42.369620842s) [noise=true , address=145.79.227.30:4786 , piece_id=7AGHJ66LWOVWFB5KE67T6VXL42KZKCA77TRTHZDBO3JHRZQUTV3A, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.539633451 (500h50m42.410716516s) [noise=true , address=77.90.31.11:4558 , piece_id=5BSZGDQHATL764TPNKFHRM7WU553KYMGZBEVJ66ZT7VPN4A43LAA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.568208433 (500h50m42.439291468s) [noise=true , address=77.90.31.21:4892 , piece_id=FOCCALPTGGXC7MIX2BBIXWALESKJQMQ4SBZ6F6655IXBFDVXZCOA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.929025190 (500h50m42.800108235s) [noise=true , address=145.79.227.40:4073 , piece_id=ZQAWPEZWMWN6U6OY5C7NV37Z4VHRI3GSL2ESQOA37GGPECCJLDZQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935653902 (500h50m42.806736947s) [noise=true , address=77.90.31.12:4622 , piece_id=RGCSHN2KH6AGECM3JU6NUFCPBON7CH46ZTWJPVHSOWYJKNS4VY5Q, seg_pos={0 0}, upload=439566]: Done uploading piece. err: <nil> Dec 26 12:19:42.935680893 (500h50m42.806763939s) [seg_pos={0 0}, upload=439566]: Segment reached optimal threshold of 14 pieces. Dec 26 12:19:42.935888318 (500h50m42.806971354s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.935897375 (500h50m42.806980431s) [noise=true , address=77.90.31.15:4715 , piece_id=HRMOYQYHGZT7JJMGJCTIMY5LMISLZDMEXKPDSDY47UCUANKPSRIA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936178431 (500h50m42.807261466s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936198199 (500h50m42.807281244s) [noise=true , address=77.90.31.17:4769 , piece_id=MKMO4BAKEXODMZSGEIRNLMPIA3SYXLS77PTBJTJNW7O32OVD7AJA, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936304120 (500h50m42.807387175s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F2dgH8YUgd1uo27B9XgTVSHFirJPQPy48bd2PQY3PzdZoUeEe): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:42.936313798 (500h50m42.807396844s) [noise=true , address=45.91.116.8:4056 , piece_id=Y26DXZB5ILOJQHCBAMHDIEOLDO37AF4MDMC4H6HLQZDIFVT53KBQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936519140 (500h50m42.807602175s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->64.49.13.169:4708: use of closed network connection Dec 26 12:19:42.936528568 (500h50m42.807611603s) [noise=true , address=64.49.13.169:4708 , piece_id=IHGX2YBF25VSQR6HYJKNBRMM5FQ72DO43QJXN4VUAAM5NONRPUHQ, seg_pos={0 0}, upload=439566]: Long tail cancellation due to optimal threshold Dec 26 12:19:42.936616866 (500h50m42.807699901s) [seg_pos={0 0}, upload=439566]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:19:42.936732155 (500h50m42.80781519s) [upload=439566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:42.936739650 (500h50m42.807822715s) [upload=439566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:43.225025927 (500h50m43.096108962s) [upload=439566]: Done uploading segments... Dec 26 12:19:43.225049321 (500h50m43.096132366s) [upload=439566]: Upload finished. err: <nil> Duration: 1m0.003424273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X......X.X.X...X.X........X..XXXXX...X.X....X..X...X..XX..] duration: 6.010277409s trace: https://jaeger.dev.storj.io/trace/1ca2c6e448115a69 FAIL (2025-12-26 12:20:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ) context deadline exceeded ======LOGS====== Dec 26 12:19:41.876746940 (500h50m41.747829985s) [upload=439567]: Starting upload Dec 26 12:19:41.876800212 (500h50m41.747883257s) [upload=439567]: Done starting upload Dec 26 12:19:41.876896925 (500h50m41.747979971s) [upload=439567]: Uploading segments... Dec 26 12:19:41.877029057 (500h50m41.748112102s) [upload=439567]: Got next segment. Inline: false Dec 26 12:19:41.877072199 (500h50m41.748155234s) [upload=439567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:19:41.877275967 (500h50m41.748359002s) [upload=439567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:19:42.123766398 (500h50m41.994849433s) [seg_pos={0 0}, upload=439567]: Begin upload segment... Dec 26 12:19:42.123844566 (500h50m41.994927601s) [seg_pos={0 0}, upload=439567]: Done begin upload segment. Dec 26 12:19:42.123902998 (500h50m41.994986033s) [noise=true , address=145.79.227.37:4565 , piece_id=UQEMY2CU46IWOPC5FJ6ZQ3L2SVFFMN2AMHWFVLC666ASBX5HZP6A, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.123935851 (500h50m41.995018886s) [noise=true , address=77.90.31.17:4773 , piece_id=7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.124479416 (500h50m41.995562461s) [noise=true , address=45.91.116.9:4261 , piece_id=CX3XDNAFOUK5CY2Y3EE2APMTQWLGGCEVQNMR7LR2CZMOD4RC3NYQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.124732478 (500h50m41.995815513s) [noise=true , address=145.79.227.36:4518 , piece_id=YGFS6XUJ5INA7VW4DQKHPA73T5VUSQPPKNO4GQUXRCDWLDTBTC7Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125057166 (500h50m41.996140201s) [noise=true , address=145.79.227.19:4078 , piece_id=MBW6DTEEMKS3JUFHY7TV5BRTYCC6MRXIB2NY2E7YLGMIPPGSDYNA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125359622 (500h50m41.996442667s) [noise=true , address=145.79.227.2:4113 , piece_id=JA3KVITDVP52CHNLGH4C6VCJQOQQFQM4J3BMWKF7OAAJWIWPBURA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125398275 (500h50m41.99648131s) [noise=true , address=145.79.227.39:4014 , piece_id=K6LK5ZSL3EUODWK7PJCDYLRDMVLFMAV7557KL6QZOGYHBNXE626Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125649193 (500h50m41.996732228s) [noise=true , address=145.79.227.5:4223 , piece_id=KDRA5HRL35MD7JYJOVTO4M66CD3PM5OLWTKF7MIQKR2YY2ZTVIOA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125827873 (500h50m41.996910918s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125953412 (500h50m41.997036457s) [noise=true , address=145.79.227.14:4840 , piece_id=OEVFYAAICKFEBD6MAVYHGLFQOZJPVZPITAOC3PJCLCHJTJ5UIXHQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126228617 (500h50m41.997311652s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126240208 (500h50m41.997323253s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126533167 (500h50m41.997616202s) [noise=true , address=45.91.116.7:4010 , piece_id=Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126594142 (500h50m41.997677177s) [noise=true , address=145.79.227.45:4231 , piece_id=IP2NOXBJ2KEYYAP3JR3XXYSQP5YDR36ILBHI6Y53L2QL7U5DLGQA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.127000256 (500h50m41.998083301s) [noise=true , address=145.79.227.4:4187 , piece_id=UWJPDDXWWGZUZYOGJ4I6NXI56GSFRI2KJWMZYJQA2HIDC7NG76XA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.128148763 (500h50m41.999231808s) [noise=true , address=45.91.116.5:4297 , piece_id=NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:43.831946619 (500h50m43.703029654s) [upload=439567]: Next returned nil segment Dec 26 12:19:43.831960697 (500h50m43.703043732s) [upload=439567]: Waiting for error group managing segments... Dec 26 12:19:44.127538768 (500h50m43.998621813s) [noise=true , address=145.79.227.5:4223 , piece_id=KDRA5HRL35MD7JYJOVTO4M66CD3PM5OLWTKF7MIQKR2YY2ZTVIOA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.129415963 (500h50m44.000498998s) [noise=true , address=145.79.227.45:4231 , piece_id=IP2NOXBJ2KEYYAP3JR3XXYSQP5YDR36ILBHI6Y53L2QL7U5DLGQA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.131062498 (500h50m44.002145533s) [noise=true , address=145.79.227.2:4113 , piece_id=JA3KVITDVP52CHNLGH4C6VCJQOQQFQM4J3BMWKF7OAAJWIWPBURA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136261638 (500h50m44.007344673s) [noise=true , address=145.79.227.39:4014 , piece_id=K6LK5ZSL3EUODWK7PJCDYLRDMVLFMAV7557KL6QZOGYHBNXE626Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136307365 (500h50m44.00739041s) [noise=true , address=145.79.227.36:4518 , piece_id=YGFS6XUJ5INA7VW4DQKHPA73T5VUSQPPKNO4GQUXRCDWLDTBTC7Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136783271 (500h50m44.007866306s) [noise=true , address=145.79.227.4:4187 , piece_id=UWJPDDXWWGZUZYOGJ4I6NXI56GSFRI2KJWMZYJQA2HIDC7NG76XA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.137438258 (500h50m44.008521293s) [noise=true , address=145.79.227.14:4840 , piece_id=OEVFYAAICKFEBD6MAVYHGLFQOZJPVZPITAOC3PJCLCHJTJ5UIXHQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.151198460 (500h50m44.022281505s) [noise=true , address=145.79.227.19:4078 , piece_id=MBW6DTEEMKS3JUFHY7TV5BRTYCC6MRXIB2NY2E7YLGMIPPGSDYNA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.160248853 (500h50m44.031331888s) [noise=true , address=145.79.227.37:4565 , piece_id=UQEMY2CU46IWOPC5FJ6ZQ3L2SVFFMN2AMHWFVLC666ASBX5HZP6A, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.461365753 (500h50m44.332448798s) [noise=true , address=45.91.116.7:4010 , piece_id=Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.500371746 (500h50m44.371454792s) [noise=true , address=45.91.116.5:4297 , piece_id=NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.523280448 (500h50m44.394363484s) [noise=true , address=77.90.31.17:4773 , piece_id=7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.662224932 (500h50m44.533307977s) [noise=true , address=45.91.116.9:4261 , piece_id=CX3XDNAFOUK5CY2Y3EE2APMTQWLGGCEVQNMR7LR2CZMOD4RC3NYQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.662279276 (500h50m44.533362311s) [seg_pos={0 0}, upload=439567]: Segment reached optimal threshold of 13 pieces. Dec 26 12:19:44.662454219 (500h50m44.533537264s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection Dec 26 12:19:44.662462445 (500h50m44.53354548s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662489166 (500h50m44.533572211s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:44.662497331 (500h50m44.533580366s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662782124 (500h50m44.533865169s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: context canceled; context canceled Dec 26 12:19:44.662792744 (500h50m44.533875789s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662812070 (500h50m44.533895116s) [seg_pos={0 0}, upload=439567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:19:44.662945435 (500h50m44.53402848s) [upload=439567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:44.662951746 (500h50m44.534034792s) [upload=439567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:44.972002824 (500h50m44.843085859s) [upload=439567]: Done uploading segments... Dec 26 12:19:44.972021861 (500h50m44.843104906s) [upload=439567]: Upload finished. err: <nil> Duration: 6.010308357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X......X.X.X...X.X........X..XXXXX...X.X....X..X...X..XX.] duration: 6.161804322s trace: https://jaeger.dev.storj.io/trace/1ca2c6e448115a69 FAIL (2025-12-26 12:20:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ) context deadline exceeded ======LOGS====== Dec 26 12:19:41.876746940 (500h50m41.747829985s) [upload=439567]: Starting upload Dec 26 12:19:41.876800212 (500h50m41.747883257s) [upload=439567]: Done starting upload Dec 26 12:19:41.876896925 (500h50m41.747979971s) [upload=439567]: Uploading segments... Dec 26 12:19:41.877029057 (500h50m41.748112102s) [upload=439567]: Got next segment. Inline: false Dec 26 12:19:41.877072199 (500h50m41.748155234s) [upload=439567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:19:41.877275967 (500h50m41.748359002s) [upload=439567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:19:42.123766398 (500h50m41.994849433s) [seg_pos={0 0}, upload=439567]: Begin upload segment... Dec 26 12:19:42.123844566 (500h50m41.994927601s) [seg_pos={0 0}, upload=439567]: Done begin upload segment. Dec 26 12:19:42.123902998 (500h50m41.994986033s) [noise=true , address=145.79.227.37:4565 , piece_id=UQEMY2CU46IWOPC5FJ6ZQ3L2SVFFMN2AMHWFVLC666ASBX5HZP6A, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.123935851 (500h50m41.995018886s) [noise=true , address=77.90.31.17:4773 , piece_id=7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.124479416 (500h50m41.995562461s) [noise=true , address=45.91.116.9:4261 , piece_id=CX3XDNAFOUK5CY2Y3EE2APMTQWLGGCEVQNMR7LR2CZMOD4RC3NYQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.124732478 (500h50m41.995815513s) [noise=true , address=145.79.227.36:4518 , piece_id=YGFS6XUJ5INA7VW4DQKHPA73T5VUSQPPKNO4GQUXRCDWLDTBTC7Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125057166 (500h50m41.996140201s) [noise=true , address=145.79.227.19:4078 , piece_id=MBW6DTEEMKS3JUFHY7TV5BRTYCC6MRXIB2NY2E7YLGMIPPGSDYNA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125359622 (500h50m41.996442667s) [noise=true , address=145.79.227.2:4113 , piece_id=JA3KVITDVP52CHNLGH4C6VCJQOQQFQM4J3BMWKF7OAAJWIWPBURA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125398275 (500h50m41.99648131s) [noise=true , address=145.79.227.39:4014 , piece_id=K6LK5ZSL3EUODWK7PJCDYLRDMVLFMAV7557KL6QZOGYHBNXE626Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125649193 (500h50m41.996732228s) [noise=true , address=145.79.227.5:4223 , piece_id=KDRA5HRL35MD7JYJOVTO4M66CD3PM5OLWTKF7MIQKR2YY2ZTVIOA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125827873 (500h50m41.996910918s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125953412 (500h50m41.997036457s) [noise=true , address=145.79.227.14:4840 , piece_id=OEVFYAAICKFEBD6MAVYHGLFQOZJPVZPITAOC3PJCLCHJTJ5UIXHQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126228617 (500h50m41.997311652s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126240208 (500h50m41.997323253s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126533167 (500h50m41.997616202s) [noise=true , address=45.91.116.7:4010 , piece_id=Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126594142 (500h50m41.997677177s) [noise=true , address=145.79.227.45:4231 , piece_id=IP2NOXBJ2KEYYAP3JR3XXYSQP5YDR36ILBHI6Y53L2QL7U5DLGQA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.127000256 (500h50m41.998083301s) [noise=true , address=145.79.227.4:4187 , piece_id=UWJPDDXWWGZUZYOGJ4I6NXI56GSFRI2KJWMZYJQA2HIDC7NG76XA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.128148763 (500h50m41.999231808s) [noise=true , address=45.91.116.5:4297 , piece_id=NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:43.831946619 (500h50m43.703029654s) [upload=439567]: Next returned nil segment Dec 26 12:19:43.831960697 (500h50m43.703043732s) [upload=439567]: Waiting for error group managing segments... Dec 26 12:19:44.127538768 (500h50m43.998621813s) [noise=true , address=145.79.227.5:4223 , piece_id=KDRA5HRL35MD7JYJOVTO4M66CD3PM5OLWTKF7MIQKR2YY2ZTVIOA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.129415963 (500h50m44.000498998s) [noise=true , address=145.79.227.45:4231 , piece_id=IP2NOXBJ2KEYYAP3JR3XXYSQP5YDR36ILBHI6Y53L2QL7U5DLGQA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.131062498 (500h50m44.002145533s) [noise=true , address=145.79.227.2:4113 , piece_id=JA3KVITDVP52CHNLGH4C6VCJQOQQFQM4J3BMWKF7OAAJWIWPBURA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136261638 (500h50m44.007344673s) [noise=true , address=145.79.227.39:4014 , piece_id=K6LK5ZSL3EUODWK7PJCDYLRDMVLFMAV7557KL6QZOGYHBNXE626Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136307365 (500h50m44.00739041s) [noise=true , address=145.79.227.36:4518 , piece_id=YGFS6XUJ5INA7VW4DQKHPA73T5VUSQPPKNO4GQUXRCDWLDTBTC7Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136783271 (500h50m44.007866306s) [noise=true , address=145.79.227.4:4187 , piece_id=UWJPDDXWWGZUZYOGJ4I6NXI56GSFRI2KJWMZYJQA2HIDC7NG76XA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.137438258 (500h50m44.008521293s) [noise=true , address=145.79.227.14:4840 , piece_id=OEVFYAAICKFEBD6MAVYHGLFQOZJPVZPITAOC3PJCLCHJTJ5UIXHQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.151198460 (500h50m44.022281505s) [noise=true , address=145.79.227.19:4078 , piece_id=MBW6DTEEMKS3JUFHY7TV5BRTYCC6MRXIB2NY2E7YLGMIPPGSDYNA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.160248853 (500h50m44.031331888s) [noise=true , address=145.79.227.37:4565 , piece_id=UQEMY2CU46IWOPC5FJ6ZQ3L2SVFFMN2AMHWFVLC666ASBX5HZP6A, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.461365753 (500h50m44.332448798s) [noise=true , address=45.91.116.7:4010 , piece_id=Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.500371746 (500h50m44.371454792s) [noise=true , address=45.91.116.5:4297 , piece_id=NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.523280448 (500h50m44.394363484s) [noise=true , address=77.90.31.17:4773 , piece_id=7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.662224932 (500h50m44.533307977s) [noise=true , address=45.91.116.9:4261 , piece_id=CX3XDNAFOUK5CY2Y3EE2APMTQWLGGCEVQNMR7LR2CZMOD4RC3NYQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.662279276 (500h50m44.533362311s) [seg_pos={0 0}, upload=439567]: Segment reached optimal threshold of 13 pieces. Dec 26 12:19:44.662454219 (500h50m44.533537264s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection Dec 26 12:19:44.662462445 (500h50m44.53354548s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662489166 (500h50m44.533572211s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:44.662497331 (500h50m44.533580366s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662782124 (500h50m44.533865169s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: context canceled; context canceled Dec 26 12:19:44.662792744 (500h50m44.533875789s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662812070 (500h50m44.533895116s) [seg_pos={0 0}, upload=439567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:19:44.662945435 (500h50m44.53402848s) [upload=439567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:44.662951746 (500h50m44.534034792s) [upload=439567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:44.972002824 (500h50m44.843085859s) [upload=439567]: Done uploading segments... Dec 26 12:19:44.972021861 (500h50m44.843104906s) [upload=439567]: Upload finished. err: <nil> Duration: 6.161831724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X......X.X.X...X.X........X..XXXXX...X.X....X..X...X..XX] duration: 30.003448251s trace: https://jaeger.dev.storj.io/trace/1ca2c6e448115a69 FAIL (2025-12-26 12:20:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ) context deadline exceeded ======LOGS====== Dec 26 12:19:41.876746940 (500h50m41.747829985s) [upload=439567]: Starting upload Dec 26 12:19:41.876800212 (500h50m41.747883257s) [upload=439567]: Done starting upload Dec 26 12:19:41.876896925 (500h50m41.747979971s) [upload=439567]: Uploading segments... Dec 26 12:19:41.877029057 (500h50m41.748112102s) [upload=439567]: Got next segment. Inline: false Dec 26 12:19:41.877072199 (500h50m41.748155234s) [upload=439567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:19:41.877275967 (500h50m41.748359002s) [upload=439567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:19:42.123766398 (500h50m41.994849433s) [seg_pos={0 0}, upload=439567]: Begin upload segment... Dec 26 12:19:42.123844566 (500h50m41.994927601s) [seg_pos={0 0}, upload=439567]: Done begin upload segment. Dec 26 12:19:42.123902998 (500h50m41.994986033s) [noise=true , address=145.79.227.37:4565 , piece_id=UQEMY2CU46IWOPC5FJ6ZQ3L2SVFFMN2AMHWFVLC666ASBX5HZP6A, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.123935851 (500h50m41.995018886s) [noise=true , address=77.90.31.17:4773 , piece_id=7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.124479416 (500h50m41.995562461s) [noise=true , address=45.91.116.9:4261 , piece_id=CX3XDNAFOUK5CY2Y3EE2APMTQWLGGCEVQNMR7LR2CZMOD4RC3NYQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.124732478 (500h50m41.995815513s) [noise=true , address=145.79.227.36:4518 , piece_id=YGFS6XUJ5INA7VW4DQKHPA73T5VUSQPPKNO4GQUXRCDWLDTBTC7Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125057166 (500h50m41.996140201s) [noise=true , address=145.79.227.19:4078 , piece_id=MBW6DTEEMKS3JUFHY7TV5BRTYCC6MRXIB2NY2E7YLGMIPPGSDYNA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125359622 (500h50m41.996442667s) [noise=true , address=145.79.227.2:4113 , piece_id=JA3KVITDVP52CHNLGH4C6VCJQOQQFQM4J3BMWKF7OAAJWIWPBURA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125398275 (500h50m41.99648131s) [noise=true , address=145.79.227.39:4014 , piece_id=K6LK5ZSL3EUODWK7PJCDYLRDMVLFMAV7557KL6QZOGYHBNXE626Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125649193 (500h50m41.996732228s) [noise=true , address=145.79.227.5:4223 , piece_id=KDRA5HRL35MD7JYJOVTO4M66CD3PM5OLWTKF7MIQKR2YY2ZTVIOA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125827873 (500h50m41.996910918s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.125953412 (500h50m41.997036457s) [noise=true , address=145.79.227.14:4840 , piece_id=OEVFYAAICKFEBD6MAVYHGLFQOZJPVZPITAOC3PJCLCHJTJ5UIXHQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126228617 (500h50m41.997311652s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126240208 (500h50m41.997323253s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126533167 (500h50m41.997616202s) [noise=true , address=45.91.116.7:4010 , piece_id=Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.126594142 (500h50m41.997677177s) [noise=true , address=145.79.227.45:4231 , piece_id=IP2NOXBJ2KEYYAP3JR3XXYSQP5YDR36ILBHI6Y53L2QL7U5DLGQA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.127000256 (500h50m41.998083301s) [noise=true , address=145.79.227.4:4187 , piece_id=UWJPDDXWWGZUZYOGJ4I6NXI56GSFRI2KJWMZYJQA2HIDC7NG76XA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:42.128148763 (500h50m41.999231808s) [noise=true , address=45.91.116.5:4297 , piece_id=NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA, seg_pos={0 0}, upload=439567]: Uploading piece... Dec 26 12:19:43.831946619 (500h50m43.703029654s) [upload=439567]: Next returned nil segment Dec 26 12:19:43.831960697 (500h50m43.703043732s) [upload=439567]: Waiting for error group managing segments... Dec 26 12:19:44.127538768 (500h50m43.998621813s) [noise=true , address=145.79.227.5:4223 , piece_id=KDRA5HRL35MD7JYJOVTO4M66CD3PM5OLWTKF7MIQKR2YY2ZTVIOA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.129415963 (500h50m44.000498998s) [noise=true , address=145.79.227.45:4231 , piece_id=IP2NOXBJ2KEYYAP3JR3XXYSQP5YDR36ILBHI6Y53L2QL7U5DLGQA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.131062498 (500h50m44.002145533s) [noise=true , address=145.79.227.2:4113 , piece_id=JA3KVITDVP52CHNLGH4C6VCJQOQQFQM4J3BMWKF7OAAJWIWPBURA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136261638 (500h50m44.007344673s) [noise=true , address=145.79.227.39:4014 , piece_id=K6LK5ZSL3EUODWK7PJCDYLRDMVLFMAV7557KL6QZOGYHBNXE626Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136307365 (500h50m44.00739041s) [noise=true , address=145.79.227.36:4518 , piece_id=YGFS6XUJ5INA7VW4DQKHPA73T5VUSQPPKNO4GQUXRCDWLDTBTC7Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.136783271 (500h50m44.007866306s) [noise=true , address=145.79.227.4:4187 , piece_id=UWJPDDXWWGZUZYOGJ4I6NXI56GSFRI2KJWMZYJQA2HIDC7NG76XA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.137438258 (500h50m44.008521293s) [noise=true , address=145.79.227.14:4840 , piece_id=OEVFYAAICKFEBD6MAVYHGLFQOZJPVZPITAOC3PJCLCHJTJ5UIXHQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.151198460 (500h50m44.022281505s) [noise=true , address=145.79.227.19:4078 , piece_id=MBW6DTEEMKS3JUFHY7TV5BRTYCC6MRXIB2NY2E7YLGMIPPGSDYNA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.160248853 (500h50m44.031331888s) [noise=true , address=145.79.227.37:4565 , piece_id=UQEMY2CU46IWOPC5FJ6ZQ3L2SVFFMN2AMHWFVLC666ASBX5HZP6A, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.461365753 (500h50m44.332448798s) [noise=true , address=45.91.116.7:4010 , piece_id=Y7C5EHLHAX64KEPXCVYWX6VKMOXBNXL776H73FWOCPTS6JCI4BTQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.500371746 (500h50m44.371454792s) [noise=true , address=45.91.116.5:4297 , piece_id=NCSULOO2ATKF42TMIGBGJ6NFMTELRPVQUYA5JRMUF4Q7OL3PHUOA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.523280448 (500h50m44.394363484s) [noise=true , address=77.90.31.17:4773 , piece_id=7OFKCFIBZ4PZ56GXPBA2B75FH2FXKTOXOHS24H4INE74KTOT2MJQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.662224932 (500h50m44.533307977s) [noise=true , address=45.91.116.9:4261 , piece_id=CX3XDNAFOUK5CY2Y3EE2APMTQWLGGCEVQNMR7LR2CZMOD4RC3NYQ, seg_pos={0 0}, upload=439567]: Done uploading piece. err: <nil> Dec 26 12:19:44.662279276 (500h50m44.533362311s) [seg_pos={0 0}, upload=439567]: Segment reached optimal threshold of 13 pieces. Dec 26 12:19:44.662454219 (500h50m44.533537264s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35910->45.91.116.14:4192: use of closed network connection Dec 26 12:19:44.662462445 (500h50m44.53354548s) [noise=true , address=45.91.116.14:4192 , piece_id=PDBUOAPACEXRG6CAH5BJWWCVFLYQ2GZMCYBKFQ5OQGXTZB26EP7Q, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662489166 (500h50m44.533572211s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:44.662497331 (500h50m44.533580366s) [noise=true , address=145.79.227.44:4194 , piece_id=UZQH3K5AOQBXIFNJGEBTC77OPL6JCIECX3TOVTGKSBWXSV4VPQ4A, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662782124 (500h50m44.533865169s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: context canceled; context canceled Dec 26 12:19:44.662792744 (500h50m44.533875789s) [noise=true , address=45.91.116.11:4120 , piece_id=OGDKBBEC2DFPDZXMAK7B6X3CTE7KVP7HG4DBJ4FJKZ6HIRXGWDXA, seg_pos={0 0}, upload=439567]: Long tail cancellation due to optimal threshold Dec 26 12:19:44.662812070 (500h50m44.533895116s) [seg_pos={0 0}, upload=439567]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:19:44.662945435 (500h50m44.53402848s) [upload=439567]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:44.662951746 (500h50m44.534034792s) [upload=439567]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:44.972002824 (500h50m44.843085859s) [upload=439567]: Done uploading segments... Dec 26 12:19:44.972021861 (500h50m44.843104906s) [upload=439567]: Upload finished. err: <nil> Duration: 30.003576506s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...........X....X............XX........X..X......X.......XXX] duration: 5.001091875s trace: https://jaeger.dev.storj.io/trace/44ed282b049beec8 FAIL (2025-12-26 12:19:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001109279s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X....X......X.X.X...X.X........X..XXXXX...X.X....X..X...X..X] duration: 30.004082842s trace: https://jaeger.dev.storj.io/trace/4928e8308d6f644d FAIL (2025-12-26 12:19:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: DQHXZWQNNVQXODDXJVVDJQIEPJGC4BLS6BW5G7V4RDHOPAY4CYLQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: KHAGHVJBTTSB62NDF673Z5OOGPOHZMBSL3U272FKLRJPBOW7GRNA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: 7GJC6ZU6AQLCCYBC2T5Z32L74G25JUWLBQPR6QGNK7ZE46Z6DKPQ) context deadline exceeded ======LOGS====== Dec 26 12:18:57.427487289 (500h49m57.298570334s) [upload=439557]: Starting upload Dec 26 12:18:57.427617798 (500h49m57.298700843s) [upload=439557]: Done starting upload Dec 26 12:18:57.427701897 (500h49m57.298784962s) [upload=439557]: Uploading segments... Dec 26 12:18:57.427962443 (500h49m57.299045478s) [upload=439557]: Got next segment. Inline: false Dec 26 12:18:57.428012619 (500h49m57.299095664s) [upload=439557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:18:57.428019442 (500h49m57.299102487s) [upload=439557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:18:57.672956123 (500h49m57.544039168s) [seg_pos={0 0}, upload=439557]: Begin upload segment... Dec 26 12:18:57.673014996 (500h49m57.544098031s) [seg_pos={0 0}, upload=439557]: Done begin upload segment. Dec 26 12:18:57.673044862 (500h49m57.544127897s) [noise=true , address=145.79.227.52:4447 , piece_id=5NKZYTIBGDHG3FAYYABTA5T7PVHXFGBSXL523YRYKF3DYNTMIMXA, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.673098184 (500h49m57.544181219s) [noise=true , address=145.79.227.37:4566 , piece_id=AGHOAXKZHHCC5IVGXR6CADM5UXW6HS7VMLZCD6VZLRVRB33SONHA, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.673741969 (500h49m57.544825014s) [noise=true , address=45.91.116.12:4228 , piece_id=7GJC6ZU6AQLCCYBC2T5Z32L74G25JUWLBQPR6QGNK7ZE46Z6DKPQ, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.674075525 (500h49m57.54515856s) [noise=true , address=145.79.227.19:4078 , piece_id=BO5BHAJKACJLUZB6UNHP3O56RZUJBA4A4DTOCHMY3THA4W26EJXA, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.674384173 (500h49m57.545467218s) [noise=true , address=145.79.227.34:4906 , piece_id=33ZNH3CF2ME2JBXG4PBBMMBA6UBLU6JW6BND6USK7VPULNR2OLXQ, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.674788532 (500h49m57.545871567s) [noise=true , address=145.79.227.22:4511 , piece_id=XGFXHU5VIJM4FRFCIYG2KBT56MJADER3NJVCJJPSHUBSR5RDD3UA, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.675102922 (500h49m57.546185957s) [noise=true , address=145.79.227.48:4302 , piece_id=DXSEHZYLKUGZMU5XPLS6PGR2GUSR2FFXULLUIGP3JHGDUBDGYDOQ, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.675416368 (500h49m57.546499403s) [noise=true , address=45.91.116.4:4940 , piece_id=CJOT7HXNOCJ7LPIPRJHXXD6SE2EUAQF5JPM7W4OMGLZVQLRDKK4Q, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.675810639 (500h49m57.546893684s) [noise=true , address=45.91.116.8:4046 , piece_id=DQHXZWQNNVQXODDXJVVDJQIEPJGC4BLS6BW5G7V4RDHOPAY4CYLQ, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.676276846 (500h49m57.547359881s) [noise=true , address=77.90.31.13:4627 , piece_id=6SB7IZ3VZHGLIXHEXPNITFREGEGXXTP7JVXRPWE5P46LPSPKQIJQ, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.676659595 (500h49m57.54774264s) [noise=true , address=145.79.227.15:4873 , piece_id=URCXJFCW6YHKFGBUED34TW5DRU7CK2TJFCWNRHDL7E6PSYC7Y3UA, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.677965142 (500h49m57.549048177s) [noise=true , address=45.91.116.14:4192 , piece_id=KHAGHVJBTTSB62NDF673Z5OOGPOHZMBSL3U272FKLRJPBOW7GRNA, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.679317196 (500h49m57.550400231s) [noise=true , address=145.79.227.11:4730 , piece_id=ER2IFMGKKQKVVOP3RJ4EGLRB77EVNDW46QCVGAINXXJNMNNSXA2Q, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.680816591 (500h49m57.551899626s) [noise=true , address=145.79.227.17:4970 , piece_id=DOB657SJK76CLSUVC4YF46QNSVBUE3STIUJ4FZOYRBBLDJX5AMTQ, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.682352716 (500h49m57.553435761s) [noise=true , address=145.79.227.5:4223 , piece_id=446VC26LCUXXYALQYNIVEZFEUVDZOQ3WXHRUFZ5426YTMIR6UK4Q, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:57.683925741 (500h49m57.555008786s) [noise=true , address=45.91.116.13:4155 , piece_id=CJVHCWUKDD5JELV2JF54VFIAK3MAW4CTEADZ7VX6OVEKKOWXJ4SQ, seg_pos={0 0}, upload=439557]: Uploading piece... Dec 26 12:18:59.394565876 (500h49m59.265648921s) [upload=439557]: Next returned nil segment Dec 26 12:18:59.394580705 (500h49m59.26566375s) [upload=439557]: Waiting for error group managing segments... Dec 26 12:18:59.671734484 (500h49m59.542817529s) [noise=true , address=145.79.227.52:4447 , piece_id=5NKZYTIBGDHG3FAYYABTA5T7PVHXFGBSXL523YRYKF3DYNTMIMXA, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:18:59.683586232 (500h49m59.554669277s) [noise=true , address=145.79.227.37:4566 , piece_id=AGHOAXKZHHCC5IVGXR6CADM5UXW6HS7VMLZCD6VZLRVRB33SONHA, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:18:59.688382845 (500h49m59.55946588s) [noise=true , address=145.79.227.17:4970 , piece_id=DOB657SJK76CLSUVC4YF46QNSVBUE3STIUJ4FZOYRBBLDJX5AMTQ, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:18:59.688740446 (500h49m59.559823481s) [noise=true , address=145.79.227.11:4730 , piece_id=ER2IFMGKKQKVVOP3RJ4EGLRB77EVNDW46QCVGAINXXJNMNNSXA2Q, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:18:59.690828181 (500h49m59.561911226s) [noise=true , address=145.79.227.22:4511 , piece_id=XGFXHU5VIJM4FRFCIYG2KBT56MJADER3NJVCJJPSHUBSR5RDD3UA, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:18:59.698978831 (500h49m59.570061876s) [noise=true , address=145.79.227.15:4873 , piece_id=URCXJFCW6YHKFGBUED34TW5DRU7CK2TJFCWNRHDL7E6PSYC7Y3UA, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:18:59.700334443 (500h49m59.571417488s) [noise=true , address=145.79.227.19:4078 , piece_id=BO5BHAJKACJLUZB6UNHP3O56RZUJBA4A4DTOCHMY3THA4W26EJXA, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:18:59.705912634 (500h49m59.576995669s) [noise=true , address=145.79.227.48:4302 , piece_id=DXSEHZYLKUGZMU5XPLS6PGR2GUSR2FFXULLUIGP3JHGDUBDGYDOQ, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:18:59.712399788 (500h49m59.583482833s) [noise=true , address=145.79.227.5:4223 , piece_id=446VC26LCUXXYALQYNIVEZFEUVDZOQ3WXHRUFZ5426YTMIR6UK4Q, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:18:59.943707017 (500h49m59.814790052s) [noise=true , address=45.91.116.12:4228 , piece_id=7GJC6ZU6AQLCCYBC2T5Z32L74G25JUWLBQPR6QGNK7ZE46Z6DKPQ, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:18:59.981207955 (500h49m59.852291s) [noise=true , address=45.91.116.14:4192 , piece_id=KHAGHVJBTTSB62NDF673Z5OOGPOHZMBSL3U272FKLRJPBOW7GRNA, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:19:00.015650119 (500h49m59.886733154s) [noise=true , address=45.91.116.8:4046 , piece_id=DQHXZWQNNVQXODDXJVVDJQIEPJGC4BLS6BW5G7V4RDHOPAY4CYLQ, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:19:00.045556917 (500h49m59.916639962s) [noise=true , address=45.91.116.13:4155 , piece_id=CJVHCWUKDD5JELV2JF54VFIAK3MAW4CTEADZ7VX6OVEKKOWXJ4SQ, seg_pos={0 0}, upload=439557]: Done uploading piece. err: <nil> Dec 26 12:19:00.045580702 (500h49m59.916663747s) [seg_pos={0 0}, upload=439557]: Segment reached optimal threshold of 13 pieces. Dec 26 12:19:00.045946138 (500h49m59.917029173s) [noise=true , address=45.91.116.4:4940 , piece_id=CJOT7HXNOCJ7LPIPRJHXXD6SE2EUAQF5JPM7W4OMGLZVQLRDKK4Q, seg_pos={0 0}, upload=439557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:44512->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:19:00.045953792 (500h49m59.917036837s) [noise=true , address=45.91.116.4:4940 , piece_id=CJOT7HXNOCJ7LPIPRJHXXD6SE2EUAQF5JPM7W4OMGLZVQLRDKK4Q, seg_pos={0 0}, upload=439557]: Long tail cancellation due to optimal threshold Dec 26 12:19:00.046021591 (500h49m59.917104626s) [noise=true , address=145.79.227.34:4906 , piece_id=33ZNH3CF2ME2JBXG4PBBMMBA6UBLU6JW6BND6USK7VPULNR2OLXQ, seg_pos={0 0}, upload=439557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:00.046040207 (500h49m59.917123242s) [noise=true , address=145.79.227.34:4906 , piece_id=33ZNH3CF2ME2JBXG4PBBMMBA6UBLU6JW6BND6USK7VPULNR2OLXQ, seg_pos={0 0}, upload=439557]: Long tail cancellation due to optimal threshold Dec 26 12:19:00.046200672 (500h49m59.917283707s) [noise=true , address=77.90.31.13:4627 , piece_id=6SB7IZ3VZHGLIXHEXPNITFREGEGXXTP7JVXRPWE5P46LPSPKQIJQ, seg_pos={0 0}, upload=439557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 26 12:19:00.046214198 (500h49m59.917297243s) [noise=true , address=77.90.31.13:4627 , piece_id=6SB7IZ3VZHGLIXHEXPNITFREGEGXXTP7JVXRPWE5P46LPSPKQIJQ, seg_pos={0 0}, upload=439557]: Long tail cancellation due to optimal threshold Dec 26 12:19:00.046237413 (500h49m59.917320458s) [seg_pos={0 0}, upload=439557]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:19:00.046358323 (500h49m59.917441368s) [upload=439557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:19:00.046367059 (500h49m59.917450094s) [upload=439557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:19:00.331386486 (500h50m0.202469531s) [upload=439557]: Done uploading segments... Dec 26 12:19:00.331409650 (500h50m0.202492695s) [upload=439557]: Upload finished. err: <nil> Duration: 30.004096999s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [............X....X............XX........X..X......X.......XX] duration: 5.001716043s trace: https://jaeger.dev.storj.io/trace/27c019c6fadf03a6 FAIL (2025-12-26 12:18:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001738828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.............X....X............XX........X..X......X.......X] duration: 5.001249715s trace: https://jaeger.dev.storj.io/trace/4b4d99c15046dea1 FAIL (2025-12-26 12:18:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001274352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X....X......X.X.X...X.X........X..XXXXX...X.X....X..X...X..] duration: 6.123379325s trace: https://jaeger.dev.storj.io/trace/6b037fe74d3c2782 FAIL (2025-12-26 12:18:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: PTFES77DP5LD6RFSQ4EKA3KQXU4CQXNXJYDNZWLMAQLBBOGI4RXA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: GEQTPGDB62XBTZXL5TWC6NUUK4T4JISG3LYMBQMIAV5JBA33XC3Q) context deadline exceeded ======LOGS====== Dec 26 12:16:49.797693564 (500h47m49.668776609s) [upload=439529]: Starting upload Dec 26 12:16:49.797776842 (500h47m49.668859907s) [upload=439529]: Done starting upload Dec 26 12:16:49.797908593 (500h47m49.668991638s) [upload=439529]: Uploading segments... Dec 26 12:16:49.798049581 (500h47m49.669132626s) [upload=439529]: Got next segment. Inline: false Dec 26 12:16:49.798095058 (500h47m49.669178093s) [upload=439529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:16:49.798100658 (500h47m49.669183703s) [upload=439529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:16:50.049982486 (500h47m49.921065531s) [seg_pos={0 0}, upload=439529]: Begin upload segment... Dec 26 12:16:50.050061055 (500h47m49.9211441s) [seg_pos={0 0}, upload=439529]: Done begin upload segment. Dec 26 12:16:50.050181464 (500h47m49.921264509s) [noise=true , address=145.79.227.11:4731 , piece_id=LPYHTCDUBEKQ33A2NEQHLBR3RYBBZ3SFW6HUXLJEUCAXPK5EDFJA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.050205610 (500h47m49.921288655s) [noise=true , address=45.91.116.13:4153 , piece_id=PTFES77DP5LD6RFSQ4EKA3KQXU4CQXNXJYDNZWLMAQLBBOGI4RXA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.050339045 (500h47m49.92142208s) [noise=true , address=145.79.227.36:4519 , piece_id=EG3K7GVECCQ34QS7BJELB5A7MVWRITT4AY6RU5322SX33O4UY2UA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.050950007 (500h47m49.922033063s) [noise=true , address=45.91.116.9:4264 , piece_id=MTUK6VRAFPWEX3FIW3FSTHKECNWF7TYQ7PRKWY4N7ZD6YFTL4OTA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.051447215 (500h47m49.92253025s) [noise=true , address=145.79.227.47:4267 , piece_id=ORSL6JUCSDQBODG2OYSYWQIPSM7LUHFUMZTFCCRY5ZJLBBEJXR7Q, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.051878295 (500h47m49.922961341s) [noise=true , address=77.90.31.14:4664 , piece_id=X3UWG66PDBI4M5W3JJXS4H5JPKG3GCAAPRWHATE2GAPYLURRMPKA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.052351467 (500h47m49.923434512s) [noise=true , address=77.90.31.11:4556 , piece_id=GEQTPGDB62XBTZXL5TWC6NUUK4T4JISG3LYMBQMIAV5JBA33XC3Q, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.052913046 (500h47m49.923996081s) [noise=true , address=145.79.227.17:4970 , piece_id=5ULTZOYU237A3FUHPD6PUDX43XLJO3H3RYQQ6MOBIT5VDYR3JY6A, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.054295970 (500h47m49.925379005s) [noise=true , address=145.79.227.29:4942 , piece_id=4Z5ZD5XQWNAIXXGLGNYX5AP3CZHJZFFM7VMWF6C4NU3DMPRMQEHQ, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.054994359 (500h47m49.926077394s) [noise=true , address=145.79.227.46:4601 , piece_id=FHV73EPDOX64SBAWBS2S724FF3HDEJWMUWU3E5ZZSP2ICQ6FLLVQ, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.055576086 (500h47m49.926659121s) [noise=true , address=145.79.227.37:4565 , piece_id=IMGPWGVQTEPZT62WPWQVJAK374IYGIU2EE6XT4SNUVIBKGQ62RTQ, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.056822630 (500h47m49.927905665s) [noise=true , address=45.91.116.12:4226 , piece_id=XEROWKDPUCBI6XIWW3K2MRAYEL2SNR4GYHVL7L3B3XO6HP6ENH3A, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.057417383 (500h47m49.928500428s) [noise=true , address=145.79.227.43:4158 , piece_id=OWDJ6GWAANU4X5UFXMD4BLG2HFXXSXKPDOEYUDXIVUUM6PTYE7MA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.058282630 (500h47m49.929365686s) [noise=true , address=145.79.227.13:4802 , piece_id=2ZGMHNRQ7WUVTVYHKIIJ75AGN7PNY3BRUMHGR6VIHS723WU2ZEFA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.059293746 (500h47m49.930376791s) [noise=true , address=45.91.116.6:4974 , piece_id=PMUJ5PXGAZFATYICMX3OOMXJYRAE4SHGPQQZUYEQA4F265DRW4BQ, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.059811983 (500h47m49.930895018s) [noise=true , address=145.79.227.4:4185 , piece_id=7DNHTQCGTCPLLVZ5HM3LKINLGXA75XSIZFRV5WLDDKVOF6JWYNRA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:51.780838005 (500h47m51.65192104s) [upload=439529]: Next returned nil segment Dec 26 12:16:51.780859705 (500h47m51.651942751s) [upload=439529]: Waiting for error group managing segments... Dec 26 12:16:52.069904127 (500h47m51.940987172s) [noise=true , address=145.79.227.46:4601 , piece_id=FHV73EPDOX64SBAWBS2S724FF3HDEJWMUWU3E5ZZSP2ICQ6FLLVQ, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.070003376 (500h47m51.941086411s) [noise=true , address=145.79.227.43:4158 , piece_id=OWDJ6GWAANU4X5UFXMD4BLG2HFXXSXKPDOEYUDXIVUUM6PTYE7MA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.071480358 (500h47m51.942563393s) [noise=true , address=145.79.227.4:4185 , piece_id=7DNHTQCGTCPLLVZ5HM3LKINLGXA75XSIZFRV5WLDDKVOF6JWYNRA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.071736435 (500h47m51.94281949s) [noise=true , address=145.79.227.13:4802 , piece_id=2ZGMHNRQ7WUVTVYHKIIJ75AGN7PNY3BRUMHGR6VIHS723WU2ZEFA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.072686405 (500h47m51.94376945s) [noise=true , address=145.79.227.29:4942 , piece_id=4Z5ZD5XQWNAIXXGLGNYX5AP3CZHJZFFM7VMWF6C4NU3DMPRMQEHQ, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.073226833 (500h47m51.944309868s) [noise=true , address=145.79.227.17:4970 , piece_id=5ULTZOYU237A3FUHPD6PUDX43XLJO3H3RYQQ6MOBIT5VDYR3JY6A, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.074224894 (500h47m51.945307939s) [noise=true , address=145.79.227.37:4565 , piece_id=IMGPWGVQTEPZT62WPWQVJAK374IYGIU2EE6XT4SNUVIBKGQ62RTQ, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.081549140 (500h47m51.952632185s) [noise=true , address=145.79.227.47:4267 , piece_id=ORSL6JUCSDQBODG2OYSYWQIPSM7LUHFUMZTFCCRY5ZJLBBEJXR7Q, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.084797386 (500h47m51.955880431s) [noise=true , address=145.79.227.36:4519 , piece_id=EG3K7GVECCQ34QS7BJELB5A7MVWRITT4AY6RU5322SX33O4UY2UA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.086425275 (500h47m51.95750832s) [noise=true , address=145.79.227.11:4731 , piece_id=LPYHTCDUBEKQ33A2NEQHLBR3RYBBZ3SFW6HUXLJEUCAXPK5EDFJA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.325414472 (500h47m52.196497508s) [noise=true , address=45.91.116.13:4153 , piece_id=PTFES77DP5LD6RFSQ4EKA3KQXU4CQXNXJYDNZWLMAQLBBOGI4RXA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.467713904 (500h47m52.338796939s) [noise=true , address=77.90.31.11:4556 , piece_id=GEQTPGDB62XBTZXL5TWC6NUUK4T4JISG3LYMBQMIAV5JBA33XC3Q, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.563818775 (500h47m52.43490182s) [noise=true , address=45.91.116.12:4226 , piece_id=XEROWKDPUCBI6XIWW3K2MRAYEL2SNR4GYHVL7L3B3XO6HP6ENH3A, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.563838353 (500h47m52.434921388s) [seg_pos={0 0}, upload=439529]: Segment reached optimal threshold of 13 pieces. Dec 26 12:16:52.564019738 (500h47m52.435102773s) [noise=true , address=45.91.116.6:4974 , piece_id=PMUJ5PXGAZFATYICMX3OOMXJYRAE4SHGPQQZUYEQA4F265DRW4BQ, seg_pos={0 0}, upload=439529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:58710->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:58710->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58710->45.91.116.6:4974: use of closed network connection Dec 26 12:16:52.564028224 (500h47m52.435111259s) [noise=true , address=45.91.116.6:4974 , piece_id=PMUJ5PXGAZFATYICMX3OOMXJYRAE4SHGPQQZUYEQA4F265DRW4BQ, seg_pos={0 0}, upload=439529]: Long tail cancellation due to optimal threshold Dec 26 12:16:52.564040488 (500h47m52.435123533s) [noise=true , address=77.90.31.14:4664 , piece_id=X3UWG66PDBI4M5W3JJXS4H5JPKG3GCAAPRWHATE2GAPYLURRMPKA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 26 12:16:52.564045427 (500h47m52.435128462s) [noise=true , address=77.90.31.14:4664 , piece_id=X3UWG66PDBI4M5W3JJXS4H5JPKG3GCAAPRWHATE2GAPYLURRMPKA, seg_pos={0 0}, upload=439529]: Long tail cancellation due to optimal threshold Dec 26 12:16:52.564138523 (500h47m52.435221559s) [noise=true , address=45.91.116.9:4264 , piece_id=MTUK6VRAFPWEX3FIW3FSTHKECNWF7TYQ7PRKWY4N7ZD6YFTL4OTA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 12:16:52.564147581 (500h47m52.435230617s) [noise=true , address=45.91.116.9:4264 , piece_id=MTUK6VRAFPWEX3FIW3FSTHKECNWF7TYQ7PRKWY4N7ZD6YFTL4OTA, seg_pos={0 0}, upload=439529]: Long tail cancellation due to optimal threshold Dec 26 12:16:52.564170775 (500h47m52.43525382s) [seg_pos={0 0}, upload=439529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:16:52.564234837 (500h47m52.435317872s) [upload=439529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:16:52.564239596 (500h47m52.435322631s) [upload=439529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:16:52.847109200 (500h47m52.718192235s) [upload=439529]: Done uploading segments... Dec 26 12:16:52.847129077 (500h47m52.718212112s) [upload=439529]: Upload finished. err: <nil> Duration: 6.123412128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X....X......X.X.X...X.X........X..XXXXX...X.X....X..X...X.] duration: 5.72615967s trace: https://jaeger.dev.storj.io/trace/6b037fe74d3c2782 FAIL (2025-12-26 12:17:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: PTFES77DP5LD6RFSQ4EKA3KQXU4CQXNXJYDNZWLMAQLBBOGI4RXA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: GEQTPGDB62XBTZXL5TWC6NUUK4T4JISG3LYMBQMIAV5JBA33XC3Q) context deadline exceeded ======LOGS====== Dec 26 12:16:49.797693564 (500h47m49.668776609s) [upload=439529]: Starting upload Dec 26 12:16:49.797776842 (500h47m49.668859907s) [upload=439529]: Done starting upload Dec 26 12:16:49.797908593 (500h47m49.668991638s) [upload=439529]: Uploading segments... Dec 26 12:16:49.798049581 (500h47m49.669132626s) [upload=439529]: Got next segment. Inline: false Dec 26 12:16:49.798095058 (500h47m49.669178093s) [upload=439529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:16:49.798100658 (500h47m49.669183703s) [upload=439529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:16:50.049982486 (500h47m49.921065531s) [seg_pos={0 0}, upload=439529]: Begin upload segment... Dec 26 12:16:50.050061055 (500h47m49.9211441s) [seg_pos={0 0}, upload=439529]: Done begin upload segment. Dec 26 12:16:50.050181464 (500h47m49.921264509s) [noise=true , address=145.79.227.11:4731 , piece_id=LPYHTCDUBEKQ33A2NEQHLBR3RYBBZ3SFW6HUXLJEUCAXPK5EDFJA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.050205610 (500h47m49.921288655s) [noise=true , address=45.91.116.13:4153 , piece_id=PTFES77DP5LD6RFSQ4EKA3KQXU4CQXNXJYDNZWLMAQLBBOGI4RXA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.050339045 (500h47m49.92142208s) [noise=true , address=145.79.227.36:4519 , piece_id=EG3K7GVECCQ34QS7BJELB5A7MVWRITT4AY6RU5322SX33O4UY2UA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.050950007 (500h47m49.922033063s) [noise=true , address=45.91.116.9:4264 , piece_id=MTUK6VRAFPWEX3FIW3FSTHKECNWF7TYQ7PRKWY4N7ZD6YFTL4OTA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.051447215 (500h47m49.92253025s) [noise=true , address=145.79.227.47:4267 , piece_id=ORSL6JUCSDQBODG2OYSYWQIPSM7LUHFUMZTFCCRY5ZJLBBEJXR7Q, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.051878295 (500h47m49.922961341s) [noise=true , address=77.90.31.14:4664 , piece_id=X3UWG66PDBI4M5W3JJXS4H5JPKG3GCAAPRWHATE2GAPYLURRMPKA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.052351467 (500h47m49.923434512s) [noise=true , address=77.90.31.11:4556 , piece_id=GEQTPGDB62XBTZXL5TWC6NUUK4T4JISG3LYMBQMIAV5JBA33XC3Q, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.052913046 (500h47m49.923996081s) [noise=true , address=145.79.227.17:4970 , piece_id=5ULTZOYU237A3FUHPD6PUDX43XLJO3H3RYQQ6MOBIT5VDYR3JY6A, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.054295970 (500h47m49.925379005s) [noise=true , address=145.79.227.29:4942 , piece_id=4Z5ZD5XQWNAIXXGLGNYX5AP3CZHJZFFM7VMWF6C4NU3DMPRMQEHQ, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.054994359 (500h47m49.926077394s) [noise=true , address=145.79.227.46:4601 , piece_id=FHV73EPDOX64SBAWBS2S724FF3HDEJWMUWU3E5ZZSP2ICQ6FLLVQ, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.055576086 (500h47m49.926659121s) [noise=true , address=145.79.227.37:4565 , piece_id=IMGPWGVQTEPZT62WPWQVJAK374IYGIU2EE6XT4SNUVIBKGQ62RTQ, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.056822630 (500h47m49.927905665s) [noise=true , address=45.91.116.12:4226 , piece_id=XEROWKDPUCBI6XIWW3K2MRAYEL2SNR4GYHVL7L3B3XO6HP6ENH3A, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.057417383 (500h47m49.928500428s) [noise=true , address=145.79.227.43:4158 , piece_id=OWDJ6GWAANU4X5UFXMD4BLG2HFXXSXKPDOEYUDXIVUUM6PTYE7MA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.058282630 (500h47m49.929365686s) [noise=true , address=145.79.227.13:4802 , piece_id=2ZGMHNRQ7WUVTVYHKIIJ75AGN7PNY3BRUMHGR6VIHS723WU2ZEFA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.059293746 (500h47m49.930376791s) [noise=true , address=45.91.116.6:4974 , piece_id=PMUJ5PXGAZFATYICMX3OOMXJYRAE4SHGPQQZUYEQA4F265DRW4BQ, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.059811983 (500h47m49.930895018s) [noise=true , address=145.79.227.4:4185 , piece_id=7DNHTQCGTCPLLVZ5HM3LKINLGXA75XSIZFRV5WLDDKVOF6JWYNRA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:51.780838005 (500h47m51.65192104s) [upload=439529]: Next returned nil segment Dec 26 12:16:51.780859705 (500h47m51.651942751s) [upload=439529]: Waiting for error group managing segments... Dec 26 12:16:52.069904127 (500h47m51.940987172s) [noise=true , address=145.79.227.46:4601 , piece_id=FHV73EPDOX64SBAWBS2S724FF3HDEJWMUWU3E5ZZSP2ICQ6FLLVQ, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.070003376 (500h47m51.941086411s) [noise=true , address=145.79.227.43:4158 , piece_id=OWDJ6GWAANU4X5UFXMD4BLG2HFXXSXKPDOEYUDXIVUUM6PTYE7MA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.071480358 (500h47m51.942563393s) [noise=true , address=145.79.227.4:4185 , piece_id=7DNHTQCGTCPLLVZ5HM3LKINLGXA75XSIZFRV5WLDDKVOF6JWYNRA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.071736435 (500h47m51.94281949s) [noise=true , address=145.79.227.13:4802 , piece_id=2ZGMHNRQ7WUVTVYHKIIJ75AGN7PNY3BRUMHGR6VIHS723WU2ZEFA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.072686405 (500h47m51.94376945s) [noise=true , address=145.79.227.29:4942 , piece_id=4Z5ZD5XQWNAIXXGLGNYX5AP3CZHJZFFM7VMWF6C4NU3DMPRMQEHQ, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.073226833 (500h47m51.944309868s) [noise=true , address=145.79.227.17:4970 , piece_id=5ULTZOYU237A3FUHPD6PUDX43XLJO3H3RYQQ6MOBIT5VDYR3JY6A, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.074224894 (500h47m51.945307939s) [noise=true , address=145.79.227.37:4565 , piece_id=IMGPWGVQTEPZT62WPWQVJAK374IYGIU2EE6XT4SNUVIBKGQ62RTQ, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.081549140 (500h47m51.952632185s) [noise=true , address=145.79.227.47:4267 , piece_id=ORSL6JUCSDQBODG2OYSYWQIPSM7LUHFUMZTFCCRY5ZJLBBEJXR7Q, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.084797386 (500h47m51.955880431s) [noise=true , address=145.79.227.36:4519 , piece_id=EG3K7GVECCQ34QS7BJELB5A7MVWRITT4AY6RU5322SX33O4UY2UA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.086425275 (500h47m51.95750832s) [noise=true , address=145.79.227.11:4731 , piece_id=LPYHTCDUBEKQ33A2NEQHLBR3RYBBZ3SFW6HUXLJEUCAXPK5EDFJA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.325414472 (500h47m52.196497508s) [noise=true , address=45.91.116.13:4153 , piece_id=PTFES77DP5LD6RFSQ4EKA3KQXU4CQXNXJYDNZWLMAQLBBOGI4RXA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.467713904 (500h47m52.338796939s) [noise=true , address=77.90.31.11:4556 , piece_id=GEQTPGDB62XBTZXL5TWC6NUUK4T4JISG3LYMBQMIAV5JBA33XC3Q, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.563818775 (500h47m52.43490182s) [noise=true , address=45.91.116.12:4226 , piece_id=XEROWKDPUCBI6XIWW3K2MRAYEL2SNR4GYHVL7L3B3XO6HP6ENH3A, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.563838353 (500h47m52.434921388s) [seg_pos={0 0}, upload=439529]: Segment reached optimal threshold of 13 pieces. Dec 26 12:16:52.564019738 (500h47m52.435102773s) [noise=true , address=45.91.116.6:4974 , piece_id=PMUJ5PXGAZFATYICMX3OOMXJYRAE4SHGPQQZUYEQA4F265DRW4BQ, seg_pos={0 0}, upload=439529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:58710->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:58710->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58710->45.91.116.6:4974: use of closed network connection Dec 26 12:16:52.564028224 (500h47m52.435111259s) [noise=true , address=45.91.116.6:4974 , piece_id=PMUJ5PXGAZFATYICMX3OOMXJYRAE4SHGPQQZUYEQA4F265DRW4BQ, seg_pos={0 0}, upload=439529]: Long tail cancellation due to optimal threshold Dec 26 12:16:52.564040488 (500h47m52.435123533s) [noise=true , address=77.90.31.14:4664 , piece_id=X3UWG66PDBI4M5W3JJXS4H5JPKG3GCAAPRWHATE2GAPYLURRMPKA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 26 12:16:52.564045427 (500h47m52.435128462s) [noise=true , address=77.90.31.14:4664 , piece_id=X3UWG66PDBI4M5W3JJXS4H5JPKG3GCAAPRWHATE2GAPYLURRMPKA, seg_pos={0 0}, upload=439529]: Long tail cancellation due to optimal threshold Dec 26 12:16:52.564138523 (500h47m52.435221559s) [noise=true , address=45.91.116.9:4264 , piece_id=MTUK6VRAFPWEX3FIW3FSTHKECNWF7TYQ7PRKWY4N7ZD6YFTL4OTA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 12:16:52.564147581 (500h47m52.435230617s) [noise=true , address=45.91.116.9:4264 , piece_id=MTUK6VRAFPWEX3FIW3FSTHKECNWF7TYQ7PRKWY4N7ZD6YFTL4OTA, seg_pos={0 0}, upload=439529]: Long tail cancellation due to optimal threshold Dec 26 12:16:52.564170775 (500h47m52.43525382s) [seg_pos={0 0}, upload=439529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:16:52.564234837 (500h47m52.435317872s) [upload=439529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:16:52.564239596 (500h47m52.435322631s) [upload=439529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:16:52.847109200 (500h47m52.718192235s) [upload=439529]: Done uploading segments... Dec 26 12:16:52.847129077 (500h47m52.718212112s) [upload=439529]: Upload finished. err: <nil> Duration: 5.726185267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X....X......X.X.X...X.X........X..XXXXX...X.X....X..X...X] duration: 30.003269522s trace: https://jaeger.dev.storj.io/trace/6b037fe74d3c2782 FAIL (2025-12-26 12:17:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: PTFES77DP5LD6RFSQ4EKA3KQXU4CQXNXJYDNZWLMAQLBBOGI4RXA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: GEQTPGDB62XBTZXL5TWC6NUUK4T4JISG3LYMBQMIAV5JBA33XC3Q) context deadline exceeded ======LOGS====== Dec 26 12:16:49.797693564 (500h47m49.668776609s) [upload=439529]: Starting upload Dec 26 12:16:49.797776842 (500h47m49.668859907s) [upload=439529]: Done starting upload Dec 26 12:16:49.797908593 (500h47m49.668991638s) [upload=439529]: Uploading segments... Dec 26 12:16:49.798049581 (500h47m49.669132626s) [upload=439529]: Got next segment. Inline: false Dec 26 12:16:49.798095058 (500h47m49.669178093s) [upload=439529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:16:49.798100658 (500h47m49.669183703s) [upload=439529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:16:50.049982486 (500h47m49.921065531s) [seg_pos={0 0}, upload=439529]: Begin upload segment... Dec 26 12:16:50.050061055 (500h47m49.9211441s) [seg_pos={0 0}, upload=439529]: Done begin upload segment. Dec 26 12:16:50.050181464 (500h47m49.921264509s) [noise=true , address=145.79.227.11:4731 , piece_id=LPYHTCDUBEKQ33A2NEQHLBR3RYBBZ3SFW6HUXLJEUCAXPK5EDFJA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.050205610 (500h47m49.921288655s) [noise=true , address=45.91.116.13:4153 , piece_id=PTFES77DP5LD6RFSQ4EKA3KQXU4CQXNXJYDNZWLMAQLBBOGI4RXA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.050339045 (500h47m49.92142208s) [noise=true , address=145.79.227.36:4519 , piece_id=EG3K7GVECCQ34QS7BJELB5A7MVWRITT4AY6RU5322SX33O4UY2UA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.050950007 (500h47m49.922033063s) [noise=true , address=45.91.116.9:4264 , piece_id=MTUK6VRAFPWEX3FIW3FSTHKECNWF7TYQ7PRKWY4N7ZD6YFTL4OTA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.051447215 (500h47m49.92253025s) [noise=true , address=145.79.227.47:4267 , piece_id=ORSL6JUCSDQBODG2OYSYWQIPSM7LUHFUMZTFCCRY5ZJLBBEJXR7Q, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.051878295 (500h47m49.922961341s) [noise=true , address=77.90.31.14:4664 , piece_id=X3UWG66PDBI4M5W3JJXS4H5JPKG3GCAAPRWHATE2GAPYLURRMPKA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.052351467 (500h47m49.923434512s) [noise=true , address=77.90.31.11:4556 , piece_id=GEQTPGDB62XBTZXL5TWC6NUUK4T4JISG3LYMBQMIAV5JBA33XC3Q, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.052913046 (500h47m49.923996081s) [noise=true , address=145.79.227.17:4970 , piece_id=5ULTZOYU237A3FUHPD6PUDX43XLJO3H3RYQQ6MOBIT5VDYR3JY6A, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.054295970 (500h47m49.925379005s) [noise=true , address=145.79.227.29:4942 , piece_id=4Z5ZD5XQWNAIXXGLGNYX5AP3CZHJZFFM7VMWF6C4NU3DMPRMQEHQ, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.054994359 (500h47m49.926077394s) [noise=true , address=145.79.227.46:4601 , piece_id=FHV73EPDOX64SBAWBS2S724FF3HDEJWMUWU3E5ZZSP2ICQ6FLLVQ, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.055576086 (500h47m49.926659121s) [noise=true , address=145.79.227.37:4565 , piece_id=IMGPWGVQTEPZT62WPWQVJAK374IYGIU2EE6XT4SNUVIBKGQ62RTQ, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.056822630 (500h47m49.927905665s) [noise=true , address=45.91.116.12:4226 , piece_id=XEROWKDPUCBI6XIWW3K2MRAYEL2SNR4GYHVL7L3B3XO6HP6ENH3A, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.057417383 (500h47m49.928500428s) [noise=true , address=145.79.227.43:4158 , piece_id=OWDJ6GWAANU4X5UFXMD4BLG2HFXXSXKPDOEYUDXIVUUM6PTYE7MA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.058282630 (500h47m49.929365686s) [noise=true , address=145.79.227.13:4802 , piece_id=2ZGMHNRQ7WUVTVYHKIIJ75AGN7PNY3BRUMHGR6VIHS723WU2ZEFA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.059293746 (500h47m49.930376791s) [noise=true , address=45.91.116.6:4974 , piece_id=PMUJ5PXGAZFATYICMX3OOMXJYRAE4SHGPQQZUYEQA4F265DRW4BQ, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:50.059811983 (500h47m49.930895018s) [noise=true , address=145.79.227.4:4185 , piece_id=7DNHTQCGTCPLLVZ5HM3LKINLGXA75XSIZFRV5WLDDKVOF6JWYNRA, seg_pos={0 0}, upload=439529]: Uploading piece... Dec 26 12:16:51.780838005 (500h47m51.65192104s) [upload=439529]: Next returned nil segment Dec 26 12:16:51.780859705 (500h47m51.651942751s) [upload=439529]: Waiting for error group managing segments... Dec 26 12:16:52.069904127 (500h47m51.940987172s) [noise=true , address=145.79.227.46:4601 , piece_id=FHV73EPDOX64SBAWBS2S724FF3HDEJWMUWU3E5ZZSP2ICQ6FLLVQ, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.070003376 (500h47m51.941086411s) [noise=true , address=145.79.227.43:4158 , piece_id=OWDJ6GWAANU4X5UFXMD4BLG2HFXXSXKPDOEYUDXIVUUM6PTYE7MA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.071480358 (500h47m51.942563393s) [noise=true , address=145.79.227.4:4185 , piece_id=7DNHTQCGTCPLLVZ5HM3LKINLGXA75XSIZFRV5WLDDKVOF6JWYNRA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.071736435 (500h47m51.94281949s) [noise=true , address=145.79.227.13:4802 , piece_id=2ZGMHNRQ7WUVTVYHKIIJ75AGN7PNY3BRUMHGR6VIHS723WU2ZEFA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.072686405 (500h47m51.94376945s) [noise=true , address=145.79.227.29:4942 , piece_id=4Z5ZD5XQWNAIXXGLGNYX5AP3CZHJZFFM7VMWF6C4NU3DMPRMQEHQ, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.073226833 (500h47m51.944309868s) [noise=true , address=145.79.227.17:4970 , piece_id=5ULTZOYU237A3FUHPD6PUDX43XLJO3H3RYQQ6MOBIT5VDYR3JY6A, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.074224894 (500h47m51.945307939s) [noise=true , address=145.79.227.37:4565 , piece_id=IMGPWGVQTEPZT62WPWQVJAK374IYGIU2EE6XT4SNUVIBKGQ62RTQ, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.081549140 (500h47m51.952632185s) [noise=true , address=145.79.227.47:4267 , piece_id=ORSL6JUCSDQBODG2OYSYWQIPSM7LUHFUMZTFCCRY5ZJLBBEJXR7Q, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.084797386 (500h47m51.955880431s) [noise=true , address=145.79.227.36:4519 , piece_id=EG3K7GVECCQ34QS7BJELB5A7MVWRITT4AY6RU5322SX33O4UY2UA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.086425275 (500h47m51.95750832s) [noise=true , address=145.79.227.11:4731 , piece_id=LPYHTCDUBEKQ33A2NEQHLBR3RYBBZ3SFW6HUXLJEUCAXPK5EDFJA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.325414472 (500h47m52.196497508s) [noise=true , address=45.91.116.13:4153 , piece_id=PTFES77DP5LD6RFSQ4EKA3KQXU4CQXNXJYDNZWLMAQLBBOGI4RXA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.467713904 (500h47m52.338796939s) [noise=true , address=77.90.31.11:4556 , piece_id=GEQTPGDB62XBTZXL5TWC6NUUK4T4JISG3LYMBQMIAV5JBA33XC3Q, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.563818775 (500h47m52.43490182s) [noise=true , address=45.91.116.12:4226 , piece_id=XEROWKDPUCBI6XIWW3K2MRAYEL2SNR4GYHVL7L3B3XO6HP6ENH3A, seg_pos={0 0}, upload=439529]: Done uploading piece. err: <nil> Dec 26 12:16:52.563838353 (500h47m52.434921388s) [seg_pos={0 0}, upload=439529]: Segment reached optimal threshold of 13 pieces. Dec 26 12:16:52.564019738 (500h47m52.435102773s) [noise=true , address=45.91.116.6:4974 , piece_id=PMUJ5PXGAZFATYICMX3OOMXJYRAE4SHGPQQZUYEQA4F265DRW4BQ, seg_pos={0 0}, upload=439529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:58710->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:58710->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58710->45.91.116.6:4974: use of closed network connection Dec 26 12:16:52.564028224 (500h47m52.435111259s) [noise=true , address=45.91.116.6:4974 , piece_id=PMUJ5PXGAZFATYICMX3OOMXJYRAE4SHGPQQZUYEQA4F265DRW4BQ, seg_pos={0 0}, upload=439529]: Long tail cancellation due to optimal threshold Dec 26 12:16:52.564040488 (500h47m52.435123533s) [noise=true , address=77.90.31.14:4664 , piece_id=X3UWG66PDBI4M5W3JJXS4H5JPKG3GCAAPRWHATE2GAPYLURRMPKA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 26 12:16:52.564045427 (500h47m52.435128462s) [noise=true , address=77.90.31.14:4664 , piece_id=X3UWG66PDBI4M5W3JJXS4H5JPKG3GCAAPRWHATE2GAPYLURRMPKA, seg_pos={0 0}, upload=439529]: Long tail cancellation due to optimal threshold Dec 26 12:16:52.564138523 (500h47m52.435221559s) [noise=true , address=45.91.116.9:4264 , piece_id=MTUK6VRAFPWEX3FIW3FSTHKECNWF7TYQ7PRKWY4N7ZD6YFTL4OTA, seg_pos={0 0}, upload=439529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 12:16:52.564147581 (500h47m52.435230617s) [noise=true , address=45.91.116.9:4264 , piece_id=MTUK6VRAFPWEX3FIW3FSTHKECNWF7TYQ7PRKWY4N7ZD6YFTL4OTA, seg_pos={0 0}, upload=439529]: Long tail cancellation due to optimal threshold Dec 26 12:16:52.564170775 (500h47m52.43525382s) [seg_pos={0 0}, upload=439529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:16:52.564234837 (500h47m52.435317872s) [upload=439529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:16:52.564239596 (500h47m52.435322631s) [upload=439529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:16:52.847109200 (500h47m52.718192235s) [upload=439529]: Done uploading segments... Dec 26 12:16:52.847129077 (500h47m52.718212112s) [upload=439529]: Upload finished. err: <nil> Duration: 30.003297515s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..............X....X............XX........X..X......X.......] duration: 3.561383441s trace: https://jaeger.dev.storj.io/trace/23358163356fbf1a FAIL (2025-12-26 12:17:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.561428997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...............X....X............XX........X..X......X......] duration: 4.851330622s trace: https://jaeger.dev.storj.io/trace/23358163356fbf1a FAIL (2025-12-26 12:16:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.851357375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...X....X......X.X.X...X.X........X..XXXXX...X.X....X..X...] duration: 5.971016781s trace: https://jaeger.dev.storj.io/trace/1c1844c2dd0143b4 FAIL (2025-12-26 12:16:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3LGG5PJD55TKBMWASWWMXWZZU5LYBSVFWMUCUAR4KXFCYHQFMZZA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: XQ7LOV3E3IOYSHVETVEUQIBILQPNYKT2UZYUFZE4FM6DLJ3ADXPA) context deadline exceeded ======LOGS====== c 26 12:13:45.232254649 (500h44m45.103337684s) [upload=439480]: Starting upload Dec 26 12:13:45.232312420 (500h44m45.103395455s) [upload=439480]: Done starting upload Dec 26 12:13:45.232428130 (500h44m45.103511175s) [upload=439480]: Uploading segments... Dec 26 12:13:45.232547116 (500h44m45.103630161s) [upload=439480]: Got next segment. Inline: false Dec 26 12:13:45.232583715 (500h44m45.10366676s) [upload=439480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:13:45.232588365 (500h44m45.1036714s) [upload=439480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:13:45.477927352 (500h44m45.349010397s) [seg_pos={0 0}, upload=439480]: Begin upload segment... Dec 26 12:13:45.477983169 (500h44m45.349066214s) [seg_pos={0 0}, upload=439480]: Done begin upload segment. Dec 26 12:13:45.478011343 (500h44m45.349094388s) [noise=true , address=145.79.227.35:4483 , piece_id=LPYLAMXLIUDNNQSFTKQ2OHQZ65XMEQHGJJ26BET45F5QRZZ6KIFA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.478022083 (500h44m45.349105118s) [noise=true , address=77.90.31.15:4700 , piece_id=XQ7LOV3E3IOYSHVETVEUQIBILQPNYKT2UZYUFZE4FM6DLJ3ADXPA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.478515652 (500h44m45.349598697s) [noise=true , address=145.79.227.3:4150 , piece_id=CO33YMIFPGDM2ATUAPIKPLLKA74CUH4LZFEBNQ3BJLH2C5IMNAKQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.479005094 (500h44m45.350088139s) [noise=true , address=45.91.116.10:4082 , piece_id=VCKFBL4SI7T4IM3RBCXTPBSKRA2KM7NR54RHBSBDFIOE52XYNHGA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.479455562 (500h44m45.350538607s) [noise=true , address=145.79.227.50:4374 , piece_id=NM475EKJFE2BSD34O4FWQHCR4AYAK3FL2ZDZ4ARAIW46SRL3HDFA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.479902614 (500h44m45.350985659s) [noise=true , address=145.79.227.34:4906 , piece_id=YYXTKLUQHUHKVKYDPIP5HA5DI5H6Z5RKD3FFRFUCGZQHQHPARHOQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.480270885 (500h44m45.35135392s) [noise=true , address=145.79.227.45:4230 , piece_id=GPWMKUXOY22V6P3QCFRRUKT7RHAVRXF3GYVI4HA2HA4VK573W32Q, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.480572519 (500h44m45.351655564s) [noise=true , address=145.79.227.4:4185 , piece_id=FCWWRPAMMVUD55JEGOOTOITVCQRP2OLEVQMGQHRALTJE3Z4XZ6EQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.480963264 (500h44m45.352046299s) [noise=true , address=145.79.227.48:4303 , piece_id=6O5SOAOMFFQCGLAMH6FP6MXCGF7AW3IDCEB7P4XLH3B2PQKDPYAQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.482488858 (500h44m45.353571903s) [noise=true , address=77.90.31.19:4844 , piece_id=W3VTA6OFW2VKMT5OV6NRWF67XSCTGYZYCHNYBT6G4MQLUO5VACRA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.484157225 (500h44m45.35524026s) [noise=true , address=145.79.227.36:4518 , piece_id=GB5EJMA4HEQ3OZRZPTHXIDIJPR2JM2GB2QBFGKSNQFXZ27VYDGJA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.484820698 (500h44m45.355903733s) [noise=true , address=145.79.227.38:4977 , piece_id=LWFE4J6SHUJHGTG4FJCCDLRMDVF63OJIMRIYYXIHRIGN6QOM4GPQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.485855318 (500h44m45.356938364s) [noise=true , address=145.79.227.12:4765 , piece_id=2KSU3DJ4YPMHGTTV4ERPSXT6ESH62BXCDDB2OYIENAE326ISLEEQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.486269086 (500h44m45.357352132s) [noise=true , address=45.91.116.6:4975 , piece_id=3LGG5PJD55TKBMWASWWMXWZZU5LYBSVFWMUCUAR4KXFCYHQFMZZA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.487477217 (500h44m45.358560262s) [noise=true , address=45.91.116.12:4226 , piece_id=XVP2IRDGYENVQX3TLZ7RMFKTFU43BO3PIWCDSGQYGJYKGV5RSVQA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.488656142 (500h44m45.359739177s) [noise=true , address=145.79.227.16:4910 , piece_id=NTCRLL3UKVE74QPWIXOZDAZS62PU3ALU6GZBLAOU2XUFPVL6PALA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:47.209954552 (500h44m47.081037597s) [upload=439480]: Next returned nil segment Dec 26 12:13:47.209981624 (500h44m47.081064679s) [upload=439480]: Waiting for error group managing segments... Dec 26 12:13:47.494435664 (500h44m47.365518699s) [noise=true , address=145.79.227.50:4374 , piece_id=NM475EKJFE2BSD34O4FWQHCR4AYAK3FL2ZDZ4ARAIW46SRL3HDFA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.496892482 (500h44m47.367975517s) [noise=true , address=145.79.227.34:4906 , piece_id=YYXTKLUQHUHKVKYDPIP5HA5DI5H6Z5RKD3FFRFUCGZQHQHPARHOQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.500452510 (500h44m47.371535556s) [noise=true , address=145.79.227.3:4150 , piece_id=CO33YMIFPGDM2ATUAPIKPLLKA74CUH4LZFEBNQ3BJLH2C5IMNAKQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.501376009 (500h44m47.372459044s) [noise=true , address=145.79.227.48:4303 , piece_id=6O5SOAOMFFQCGLAMH6FP6MXCGF7AW3IDCEB7P4XLH3B2PQKDPYAQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.502407073 (500h44m47.373490118s) [noise=true , address=145.79.227.45:4230 , piece_id=GPWMKUXOY22V6P3QCFRRUKT7RHAVRXF3GYVI4HA2HA4VK573W32Q, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.507406572 (500h44m47.378489607s) [noise=true , address=145.79.227.4:4185 , piece_id=FCWWRPAMMVUD55JEGOOTOITVCQRP2OLEVQMGQHRALTJE3Z4XZ6EQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.507503537 (500h44m47.378586572s) [noise=true , address=145.79.227.35:4483 , piece_id=LPYLAMXLIUDNNQSFTKQ2OHQZ65XMEQHGJJ26BET45F5QRZZ6KIFA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.510095452 (500h44m47.381178487s) [noise=true , address=145.79.227.38:4977 , piece_id=LWFE4J6SHUJHGTG4FJCCDLRMDVF63OJIMRIYYXIHRIGN6QOM4GPQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.510758976 (500h44m47.381842031s) [noise=true , address=145.79.227.36:4518 , piece_id=GB5EJMA4HEQ3OZRZPTHXIDIJPR2JM2GB2QBFGKSNQFXZ27VYDGJA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.512636902 (500h44m47.383719937s) [noise=true , address=145.79.227.16:4910 , piece_id=NTCRLL3UKVE74QPWIXOZDAZS62PU3ALU6GZBLAOU2XUFPVL6PALA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.517959307 (500h44m47.389042342s) [noise=true , address=145.79.227.12:4765 , piece_id=2KSU3DJ4YPMHGTTV4ERPSXT6ESH62BXCDDB2OYIENAE326ISLEEQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.799208164 (500h44m47.670291209s) [noise=true , address=45.91.116.6:4975 , piece_id=3LGG5PJD55TKBMWASWWMXWZZU5LYBSVFWMUCUAR4KXFCYHQFMZZA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.823503727 (500h44m47.694586772s) [noise=true , address=77.90.31.15:4700 , piece_id=XQ7LOV3E3IOYSHVETVEUQIBILQPNYKT2UZYUFZE4FM6DLJ3ADXPA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.823528834 (500h44m47.694611869s) [seg_pos={0 0}, upload=439480]: Segment reached optimal threshold of 13 pieces. Dec 26 12:13:47.823719397 (500h44m47.694802432s) [noise=true , address=45.91.116.10:4082 , piece_id=VCKFBL4SI7T4IM3RBCXTPBSKRA2KM7NR54RHBSBDFIOE52XYNHGA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 26 12:13:47.823729867 (500h44m47.694812922s) [noise=true , address=45.91.116.10:4082 , piece_id=VCKFBL4SI7T4IM3RBCXTPBSKRA2KM7NR54RHBSBDFIOE52XYNHGA, seg_pos={0 0}, upload=439480]: Long tail cancellation due to optimal threshold Dec 26 12:13:47.823775204 (500h44m47.694858239s) [noise=true , address=77.90.31.19:4844 , piece_id=W3VTA6OFW2VKMT5OV6NRWF67XSCTGYZYCHNYBT6G4MQLUO5VACRA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:43582->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:43582->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43582->77.90.31.19:4844: use of closed network connection Dec 26 12:13:47.823783219 (500h44m47.694866254s) [noise=true , address=77.90.31.19:4844 , piece_id=W3VTA6OFW2VKMT5OV6NRWF67XSCTGYZYCHNYBT6G4MQLUO5VACRA, seg_pos={0 0}, upload=439480]: Long tail cancellation due to optimal threshold Dec 26 12:13:47.823964143 (500h44m47.695047198s) [noise=true , address=45.91.116.12:4226 , piece_id=XVP2IRDGYENVQX3TLZ7RMFKTFU43BO3PIWCDSGQYGJYKGV5RSVQA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 12:13:47.823973531 (500h44m47.695056576s) [noise=true , address=45.91.116.12:4226 , piece_id=XVP2IRDGYENVQX3TLZ7RMFKTFU43BO3PIWCDSGQYGJYKGV5RSVQA, seg_pos={0 0}, upload=439480]: Long tail cancellation due to optimal threshold Dec 26 12:13:47.823995603 (500h44m47.695078648s) [seg_pos={0 0}, upload=439480]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:13:47.824073261 (500h44m47.695156296s) [upload=439480]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:13:47.824079472 (500h44m47.695162507s) [upload=439480]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:13:48.108768200 (500h44m47.979851245s) [upload=439480]: Done uploading segments... Dec 26 12:13:48.108793547 (500h44m47.979876592s) [upload=439480]: Upload finished. err: <nil> Duration: 5.971045939s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [................X....X............XX........X..X......X.....] duration: 5.000872389s trace: https://jaeger.dev.storj.io/trace/23358163356fbf1a FAIL (2025-12-26 12:15:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000902836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X...X....X......X.X.X...X.X........X..XXXXX...X.X....X..X..] duration: 5.843966745s trace: https://jaeger.dev.storj.io/trace/1c1844c2dd0143b4 FAIL (2025-12-26 12:15:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3LGG5PJD55TKBMWASWWMXWZZU5LYBSVFWMUCUAR4KXFCYHQFMZZA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: XQ7LOV3E3IOYSHVETVEUQIBILQPNYKT2UZYUFZE4FM6DLJ3ADXPA) context deadline exceeded ======LOGS====== c 26 12:13:45.232254649 (500h44m45.103337684s) [upload=439480]: Starting upload Dec 26 12:13:45.232312420 (500h44m45.103395455s) [upload=439480]: Done starting upload Dec 26 12:13:45.232428130 (500h44m45.103511175s) [upload=439480]: Uploading segments... Dec 26 12:13:45.232547116 (500h44m45.103630161s) [upload=439480]: Got next segment. Inline: false Dec 26 12:13:45.232583715 (500h44m45.10366676s) [upload=439480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:13:45.232588365 (500h44m45.1036714s) [upload=439480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:13:45.477927352 (500h44m45.349010397s) [seg_pos={0 0}, upload=439480]: Begin upload segment... Dec 26 12:13:45.477983169 (500h44m45.349066214s) [seg_pos={0 0}, upload=439480]: Done begin upload segment. Dec 26 12:13:45.478011343 (500h44m45.349094388s) [noise=true , address=145.79.227.35:4483 , piece_id=LPYLAMXLIUDNNQSFTKQ2OHQZ65XMEQHGJJ26BET45F5QRZZ6KIFA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.478022083 (500h44m45.349105118s) [noise=true , address=77.90.31.15:4700 , piece_id=XQ7LOV3E3IOYSHVETVEUQIBILQPNYKT2UZYUFZE4FM6DLJ3ADXPA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.478515652 (500h44m45.349598697s) [noise=true , address=145.79.227.3:4150 , piece_id=CO33YMIFPGDM2ATUAPIKPLLKA74CUH4LZFEBNQ3BJLH2C5IMNAKQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.479005094 (500h44m45.350088139s) [noise=true , address=45.91.116.10:4082 , piece_id=VCKFBL4SI7T4IM3RBCXTPBSKRA2KM7NR54RHBSBDFIOE52XYNHGA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.479455562 (500h44m45.350538607s) [noise=true , address=145.79.227.50:4374 , piece_id=NM475EKJFE2BSD34O4FWQHCR4AYAK3FL2ZDZ4ARAIW46SRL3HDFA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.479902614 (500h44m45.350985659s) [noise=true , address=145.79.227.34:4906 , piece_id=YYXTKLUQHUHKVKYDPIP5HA5DI5H6Z5RKD3FFRFUCGZQHQHPARHOQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.480270885 (500h44m45.35135392s) [noise=true , address=145.79.227.45:4230 , piece_id=GPWMKUXOY22V6P3QCFRRUKT7RHAVRXF3GYVI4HA2HA4VK573W32Q, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.480572519 (500h44m45.351655564s) [noise=true , address=145.79.227.4:4185 , piece_id=FCWWRPAMMVUD55JEGOOTOITVCQRP2OLEVQMGQHRALTJE3Z4XZ6EQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.480963264 (500h44m45.352046299s) [noise=true , address=145.79.227.48:4303 , piece_id=6O5SOAOMFFQCGLAMH6FP6MXCGF7AW3IDCEB7P4XLH3B2PQKDPYAQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.482488858 (500h44m45.353571903s) [noise=true , address=77.90.31.19:4844 , piece_id=W3VTA6OFW2VKMT5OV6NRWF67XSCTGYZYCHNYBT6G4MQLUO5VACRA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.484157225 (500h44m45.35524026s) [noise=true , address=145.79.227.36:4518 , piece_id=GB5EJMA4HEQ3OZRZPTHXIDIJPR2JM2GB2QBFGKSNQFXZ27VYDGJA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.484820698 (500h44m45.355903733s) [noise=true , address=145.79.227.38:4977 , piece_id=LWFE4J6SHUJHGTG4FJCCDLRMDVF63OJIMRIYYXIHRIGN6QOM4GPQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.485855318 (500h44m45.356938364s) [noise=true , address=145.79.227.12:4765 , piece_id=2KSU3DJ4YPMHGTTV4ERPSXT6ESH62BXCDDB2OYIENAE326ISLEEQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.486269086 (500h44m45.357352132s) [noise=true , address=45.91.116.6:4975 , piece_id=3LGG5PJD55TKBMWASWWMXWZZU5LYBSVFWMUCUAR4KXFCYHQFMZZA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.487477217 (500h44m45.358560262s) [noise=true , address=45.91.116.12:4226 , piece_id=XVP2IRDGYENVQX3TLZ7RMFKTFU43BO3PIWCDSGQYGJYKGV5RSVQA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.488656142 (500h44m45.359739177s) [noise=true , address=145.79.227.16:4910 , piece_id=NTCRLL3UKVE74QPWIXOZDAZS62PU3ALU6GZBLAOU2XUFPVL6PALA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:47.209954552 (500h44m47.081037597s) [upload=439480]: Next returned nil segment Dec 26 12:13:47.209981624 (500h44m47.081064679s) [upload=439480]: Waiting for error group managing segments... Dec 26 12:13:47.494435664 (500h44m47.365518699s) [noise=true , address=145.79.227.50:4374 , piece_id=NM475EKJFE2BSD34O4FWQHCR4AYAK3FL2ZDZ4ARAIW46SRL3HDFA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.496892482 (500h44m47.367975517s) [noise=true , address=145.79.227.34:4906 , piece_id=YYXTKLUQHUHKVKYDPIP5HA5DI5H6Z5RKD3FFRFUCGZQHQHPARHOQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.500452510 (500h44m47.371535556s) [noise=true , address=145.79.227.3:4150 , piece_id=CO33YMIFPGDM2ATUAPIKPLLKA74CUH4LZFEBNQ3BJLH2C5IMNAKQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.501376009 (500h44m47.372459044s) [noise=true , address=145.79.227.48:4303 , piece_id=6O5SOAOMFFQCGLAMH6FP6MXCGF7AW3IDCEB7P4XLH3B2PQKDPYAQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.502407073 (500h44m47.373490118s) [noise=true , address=145.79.227.45:4230 , piece_id=GPWMKUXOY22V6P3QCFRRUKT7RHAVRXF3GYVI4HA2HA4VK573W32Q, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.507406572 (500h44m47.378489607s) [noise=true , address=145.79.227.4:4185 , piece_id=FCWWRPAMMVUD55JEGOOTOITVCQRP2OLEVQMGQHRALTJE3Z4XZ6EQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.507503537 (500h44m47.378586572s) [noise=true , address=145.79.227.35:4483 , piece_id=LPYLAMXLIUDNNQSFTKQ2OHQZ65XMEQHGJJ26BET45F5QRZZ6KIFA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.510095452 (500h44m47.381178487s) [noise=true , address=145.79.227.38:4977 , piece_id=LWFE4J6SHUJHGTG4FJCCDLRMDVF63OJIMRIYYXIHRIGN6QOM4GPQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.510758976 (500h44m47.381842031s) [noise=true , address=145.79.227.36:4518 , piece_id=GB5EJMA4HEQ3OZRZPTHXIDIJPR2JM2GB2QBFGKSNQFXZ27VYDGJA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.512636902 (500h44m47.383719937s) [noise=true , address=145.79.227.16:4910 , piece_id=NTCRLL3UKVE74QPWIXOZDAZS62PU3ALU6GZBLAOU2XUFPVL6PALA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.517959307 (500h44m47.389042342s) [noise=true , address=145.79.227.12:4765 , piece_id=2KSU3DJ4YPMHGTTV4ERPSXT6ESH62BXCDDB2OYIENAE326ISLEEQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.799208164 (500h44m47.670291209s) [noise=true , address=45.91.116.6:4975 , piece_id=3LGG5PJD55TKBMWASWWMXWZZU5LYBSVFWMUCUAR4KXFCYHQFMZZA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.823503727 (500h44m47.694586772s) [noise=true , address=77.90.31.15:4700 , piece_id=XQ7LOV3E3IOYSHVETVEUQIBILQPNYKT2UZYUFZE4FM6DLJ3ADXPA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.823528834 (500h44m47.694611869s) [seg_pos={0 0}, upload=439480]: Segment reached optimal threshold of 13 pieces. Dec 26 12:13:47.823719397 (500h44m47.694802432s) [noise=true , address=45.91.116.10:4082 , piece_id=VCKFBL4SI7T4IM3RBCXTPBSKRA2KM7NR54RHBSBDFIOE52XYNHGA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 26 12:13:47.823729867 (500h44m47.694812922s) [noise=true , address=45.91.116.10:4082 , piece_id=VCKFBL4SI7T4IM3RBCXTPBSKRA2KM7NR54RHBSBDFIOE52XYNHGA, seg_pos={0 0}, upload=439480]: Long tail cancellation due to optimal threshold Dec 26 12:13:47.823775204 (500h44m47.694858239s) [noise=true , address=77.90.31.19:4844 , piece_id=W3VTA6OFW2VKMT5OV6NRWF67XSCTGYZYCHNYBT6G4MQLUO5VACRA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:43582->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:43582->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43582->77.90.31.19:4844: use of closed network connection Dec 26 12:13:47.823783219 (500h44m47.694866254s) [noise=true , address=77.90.31.19:4844 , piece_id=W3VTA6OFW2VKMT5OV6NRWF67XSCTGYZYCHNYBT6G4MQLUO5VACRA, seg_pos={0 0}, upload=439480]: Long tail cancellation due to optimal threshold Dec 26 12:13:47.823964143 (500h44m47.695047198s) [noise=true , address=45.91.116.12:4226 , piece_id=XVP2IRDGYENVQX3TLZ7RMFKTFU43BO3PIWCDSGQYGJYKGV5RSVQA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 12:13:47.823973531 (500h44m47.695056576s) [noise=true , address=45.91.116.12:4226 , piece_id=XVP2IRDGYENVQX3TLZ7RMFKTFU43BO3PIWCDSGQYGJYKGV5RSVQA, seg_pos={0 0}, upload=439480]: Long tail cancellation due to optimal threshold Dec 26 12:13:47.823995603 (500h44m47.695078648s) [seg_pos={0 0}, upload=439480]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:13:47.824073261 (500h44m47.695156296s) [upload=439480]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:13:47.824079472 (500h44m47.695162507s) [upload=439480]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:13:48.108768200 (500h44m47.979851245s) [upload=439480]: Done uploading segments... Dec 26 12:13:48.108793547 (500h44m47.979876592s) [upload=439480]: Upload finished. err: <nil> Duration: 5.843996512s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.358354553s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 12:15:06 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.358432491s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.................X....X............XX........X..X......X....] duration: 3.530353214s trace: https://jaeger.dev.storj.io/trace/23358163356fbf1a FAIL (2025-12-26 12:15:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.530380476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X...X....X......X.X.X...X.X........X..XXXXX...X.X....X..X.] duration: 5.743195219s trace: https://jaeger.dev.storj.io/trace/1c1844c2dd0143b4 FAIL (2025-12-26 12:14:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3LGG5PJD55TKBMWASWWMXWZZU5LYBSVFWMUCUAR4KXFCYHQFMZZA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: XQ7LOV3E3IOYSHVETVEUQIBILQPNYKT2UZYUFZE4FM6DLJ3ADXPA) context deadline exceeded ======LOGS====== c 26 12:13:45.232254649 (500h44m45.103337684s) [upload=439480]: Starting upload Dec 26 12:13:45.232312420 (500h44m45.103395455s) [upload=439480]: Done starting upload Dec 26 12:13:45.232428130 (500h44m45.103511175s) [upload=439480]: Uploading segments... Dec 26 12:13:45.232547116 (500h44m45.103630161s) [upload=439480]: Got next segment. Inline: false Dec 26 12:13:45.232583715 (500h44m45.10366676s) [upload=439480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:13:45.232588365 (500h44m45.1036714s) [upload=439480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:13:45.477927352 (500h44m45.349010397s) [seg_pos={0 0}, upload=439480]: Begin upload segment... Dec 26 12:13:45.477983169 (500h44m45.349066214s) [seg_pos={0 0}, upload=439480]: Done begin upload segment. Dec 26 12:13:45.478011343 (500h44m45.349094388s) [noise=true , address=145.79.227.35:4483 , piece_id=LPYLAMXLIUDNNQSFTKQ2OHQZ65XMEQHGJJ26BET45F5QRZZ6KIFA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.478022083 (500h44m45.349105118s) [noise=true , address=77.90.31.15:4700 , piece_id=XQ7LOV3E3IOYSHVETVEUQIBILQPNYKT2UZYUFZE4FM6DLJ3ADXPA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.478515652 (500h44m45.349598697s) [noise=true , address=145.79.227.3:4150 , piece_id=CO33YMIFPGDM2ATUAPIKPLLKA74CUH4LZFEBNQ3BJLH2C5IMNAKQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.479005094 (500h44m45.350088139s) [noise=true , address=45.91.116.10:4082 , piece_id=VCKFBL4SI7T4IM3RBCXTPBSKRA2KM7NR54RHBSBDFIOE52XYNHGA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.479455562 (500h44m45.350538607s) [noise=true , address=145.79.227.50:4374 , piece_id=NM475EKJFE2BSD34O4FWQHCR4AYAK3FL2ZDZ4ARAIW46SRL3HDFA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.479902614 (500h44m45.350985659s) [noise=true , address=145.79.227.34:4906 , piece_id=YYXTKLUQHUHKVKYDPIP5HA5DI5H6Z5RKD3FFRFUCGZQHQHPARHOQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.480270885 (500h44m45.35135392s) [noise=true , address=145.79.227.45:4230 , piece_id=GPWMKUXOY22V6P3QCFRRUKT7RHAVRXF3GYVI4HA2HA4VK573W32Q, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.480572519 (500h44m45.351655564s) [noise=true , address=145.79.227.4:4185 , piece_id=FCWWRPAMMVUD55JEGOOTOITVCQRP2OLEVQMGQHRALTJE3Z4XZ6EQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.480963264 (500h44m45.352046299s) [noise=true , address=145.79.227.48:4303 , piece_id=6O5SOAOMFFQCGLAMH6FP6MXCGF7AW3IDCEB7P4XLH3B2PQKDPYAQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.482488858 (500h44m45.353571903s) [noise=true , address=77.90.31.19:4844 , piece_id=W3VTA6OFW2VKMT5OV6NRWF67XSCTGYZYCHNYBT6G4MQLUO5VACRA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.484157225 (500h44m45.35524026s) [noise=true , address=145.79.227.36:4518 , piece_id=GB5EJMA4HEQ3OZRZPTHXIDIJPR2JM2GB2QBFGKSNQFXZ27VYDGJA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.484820698 (500h44m45.355903733s) [noise=true , address=145.79.227.38:4977 , piece_id=LWFE4J6SHUJHGTG4FJCCDLRMDVF63OJIMRIYYXIHRIGN6QOM4GPQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.485855318 (500h44m45.356938364s) [noise=true , address=145.79.227.12:4765 , piece_id=2KSU3DJ4YPMHGTTV4ERPSXT6ESH62BXCDDB2OYIENAE326ISLEEQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.486269086 (500h44m45.357352132s) [noise=true , address=45.91.116.6:4975 , piece_id=3LGG5PJD55TKBMWASWWMXWZZU5LYBSVFWMUCUAR4KXFCYHQFMZZA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.487477217 (500h44m45.358560262s) [noise=true , address=45.91.116.12:4226 , piece_id=XVP2IRDGYENVQX3TLZ7RMFKTFU43BO3PIWCDSGQYGJYKGV5RSVQA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.488656142 (500h44m45.359739177s) [noise=true , address=145.79.227.16:4910 , piece_id=NTCRLL3UKVE74QPWIXOZDAZS62PU3ALU6GZBLAOU2XUFPVL6PALA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:47.209954552 (500h44m47.081037597s) [upload=439480]: Next returned nil segment Dec 26 12:13:47.209981624 (500h44m47.081064679s) [upload=439480]: Waiting for error group managing segments... Dec 26 12:13:47.494435664 (500h44m47.365518699s) [noise=true , address=145.79.227.50:4374 , piece_id=NM475EKJFE2BSD34O4FWQHCR4AYAK3FL2ZDZ4ARAIW46SRL3HDFA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.496892482 (500h44m47.367975517s) [noise=true , address=145.79.227.34:4906 , piece_id=YYXTKLUQHUHKVKYDPIP5HA5DI5H6Z5RKD3FFRFUCGZQHQHPARHOQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.500452510 (500h44m47.371535556s) [noise=true , address=145.79.227.3:4150 , piece_id=CO33YMIFPGDM2ATUAPIKPLLKA74CUH4LZFEBNQ3BJLH2C5IMNAKQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.501376009 (500h44m47.372459044s) [noise=true , address=145.79.227.48:4303 , piece_id=6O5SOAOMFFQCGLAMH6FP6MXCGF7AW3IDCEB7P4XLH3B2PQKDPYAQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.502407073 (500h44m47.373490118s) [noise=true , address=145.79.227.45:4230 , piece_id=GPWMKUXOY22V6P3QCFRRUKT7RHAVRXF3GYVI4HA2HA4VK573W32Q, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.507406572 (500h44m47.378489607s) [noise=true , address=145.79.227.4:4185 , piece_id=FCWWRPAMMVUD55JEGOOTOITVCQRP2OLEVQMGQHRALTJE3Z4XZ6EQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.507503537 (500h44m47.378586572s) [noise=true , address=145.79.227.35:4483 , piece_id=LPYLAMXLIUDNNQSFTKQ2OHQZ65XMEQHGJJ26BET45F5QRZZ6KIFA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.510095452 (500h44m47.381178487s) [noise=true , address=145.79.227.38:4977 , piece_id=LWFE4J6SHUJHGTG4FJCCDLRMDVF63OJIMRIYYXIHRIGN6QOM4GPQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.510758976 (500h44m47.381842031s) [noise=true , address=145.79.227.36:4518 , piece_id=GB5EJMA4HEQ3OZRZPTHXIDIJPR2JM2GB2QBFGKSNQFXZ27VYDGJA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.512636902 (500h44m47.383719937s) [noise=true , address=145.79.227.16:4910 , piece_id=NTCRLL3UKVE74QPWIXOZDAZS62PU3ALU6GZBLAOU2XUFPVL6PALA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.517959307 (500h44m47.389042342s) [noise=true , address=145.79.227.12:4765 , piece_id=2KSU3DJ4YPMHGTTV4ERPSXT6ESH62BXCDDB2OYIENAE326ISLEEQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.799208164 (500h44m47.670291209s) [noise=true , address=45.91.116.6:4975 , piece_id=3LGG5PJD55TKBMWASWWMXWZZU5LYBSVFWMUCUAR4KXFCYHQFMZZA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.823503727 (500h44m47.694586772s) [noise=true , address=77.90.31.15:4700 , piece_id=XQ7LOV3E3IOYSHVETVEUQIBILQPNYKT2UZYUFZE4FM6DLJ3ADXPA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.823528834 (500h44m47.694611869s) [seg_pos={0 0}, upload=439480]: Segment reached optimal threshold of 13 pieces. Dec 26 12:13:47.823719397 (500h44m47.694802432s) [noise=true , address=45.91.116.10:4082 , piece_id=VCKFBL4SI7T4IM3RBCXTPBSKRA2KM7NR54RHBSBDFIOE52XYNHGA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 26 12:13:47.823729867 (500h44m47.694812922s) [noise=true , address=45.91.116.10:4082 , piece_id=VCKFBL4SI7T4IM3RBCXTPBSKRA2KM7NR54RHBSBDFIOE52XYNHGA, seg_pos={0 0}, upload=439480]: Long tail cancellation due to optimal threshold Dec 26 12:13:47.823775204 (500h44m47.694858239s) [noise=true , address=77.90.31.19:4844 , piece_id=W3VTA6OFW2VKMT5OV6NRWF67XSCTGYZYCHNYBT6G4MQLUO5VACRA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:43582->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:43582->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43582->77.90.31.19:4844: use of closed network connection Dec 26 12:13:47.823783219 (500h44m47.694866254s) [noise=true , address=77.90.31.19:4844 , piece_id=W3VTA6OFW2VKMT5OV6NRWF67XSCTGYZYCHNYBT6G4MQLUO5VACRA, seg_pos={0 0}, upload=439480]: Long tail cancellation due to optimal threshold Dec 26 12:13:47.823964143 (500h44m47.695047198s) [noise=true , address=45.91.116.12:4226 , piece_id=XVP2IRDGYENVQX3TLZ7RMFKTFU43BO3PIWCDSGQYGJYKGV5RSVQA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 12:13:47.823973531 (500h44m47.695056576s) [noise=true , address=45.91.116.12:4226 , piece_id=XVP2IRDGYENVQX3TLZ7RMFKTFU43BO3PIWCDSGQYGJYKGV5RSVQA, seg_pos={0 0}, upload=439480]: Long tail cancellation due to optimal threshold Dec 26 12:13:47.823995603 (500h44m47.695078648s) [seg_pos={0 0}, upload=439480]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:13:47.824073261 (500h44m47.695156296s) [upload=439480]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:13:47.824079472 (500h44m47.695162507s) [upload=439480]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:13:48.108768200 (500h44m47.979851245s) [upload=439480]: Done uploading segments... Dec 26 12:13:48.108793547 (500h44m47.979876592s) [upload=439480]: Upload finished. err: <nil> Duration: 5.743227781s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..................X....X............XX........X..X......X...] duration: 4.218945303s trace: https://jaeger.dev.storj.io/trace/23358163356fbf1a FAIL (2025-12-26 12:14:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.218972665s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...................X....X............XX........X..X......X..] duration: 3.913268319s trace: https://jaeger.dev.storj.io/trace/23358163356fbf1a FAIL (2025-12-26 12:14:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.913295894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.373746622s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 12:14:15 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.373778412s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.X...X....X......X.X.X...X.X........X..XXXXX...X.X....X..X] duration: 30.004149118s trace: https://jaeger.dev.storj.io/trace/1c1844c2dd0143b4 FAIL (2025-12-26 12:14:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 3LGG5PJD55TKBMWASWWMXWZZU5LYBSVFWMUCUAR4KXFCYHQFMZZA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: XQ7LOV3E3IOYSHVETVEUQIBILQPNYKT2UZYUFZE4FM6DLJ3ADXPA) context deadline exceeded ======LOGS====== c 26 12:13:45.232254649 (500h44m45.103337684s) [upload=439480]: Starting upload Dec 26 12:13:45.232312420 (500h44m45.103395455s) [upload=439480]: Done starting upload Dec 26 12:13:45.232428130 (500h44m45.103511175s) [upload=439480]: Uploading segments... Dec 26 12:13:45.232547116 (500h44m45.103630161s) [upload=439480]: Got next segment. Inline: false Dec 26 12:13:45.232583715 (500h44m45.10366676s) [upload=439480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:13:45.232588365 (500h44m45.1036714s) [upload=439480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:13:45.477927352 (500h44m45.349010397s) [seg_pos={0 0}, upload=439480]: Begin upload segment... Dec 26 12:13:45.477983169 (500h44m45.349066214s) [seg_pos={0 0}, upload=439480]: Done begin upload segment. Dec 26 12:13:45.478011343 (500h44m45.349094388s) [noise=true , address=145.79.227.35:4483 , piece_id=LPYLAMXLIUDNNQSFTKQ2OHQZ65XMEQHGJJ26BET45F5QRZZ6KIFA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.478022083 (500h44m45.349105118s) [noise=true , address=77.90.31.15:4700 , piece_id=XQ7LOV3E3IOYSHVETVEUQIBILQPNYKT2UZYUFZE4FM6DLJ3ADXPA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.478515652 (500h44m45.349598697s) [noise=true , address=145.79.227.3:4150 , piece_id=CO33YMIFPGDM2ATUAPIKPLLKA74CUH4LZFEBNQ3BJLH2C5IMNAKQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.479005094 (500h44m45.350088139s) [noise=true , address=45.91.116.10:4082 , piece_id=VCKFBL4SI7T4IM3RBCXTPBSKRA2KM7NR54RHBSBDFIOE52XYNHGA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.479455562 (500h44m45.350538607s) [noise=true , address=145.79.227.50:4374 , piece_id=NM475EKJFE2BSD34O4FWQHCR4AYAK3FL2ZDZ4ARAIW46SRL3HDFA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.479902614 (500h44m45.350985659s) [noise=true , address=145.79.227.34:4906 , piece_id=YYXTKLUQHUHKVKYDPIP5HA5DI5H6Z5RKD3FFRFUCGZQHQHPARHOQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.480270885 (500h44m45.35135392s) [noise=true , address=145.79.227.45:4230 , piece_id=GPWMKUXOY22V6P3QCFRRUKT7RHAVRXF3GYVI4HA2HA4VK573W32Q, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.480572519 (500h44m45.351655564s) [noise=true , address=145.79.227.4:4185 , piece_id=FCWWRPAMMVUD55JEGOOTOITVCQRP2OLEVQMGQHRALTJE3Z4XZ6EQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.480963264 (500h44m45.352046299s) [noise=true , address=145.79.227.48:4303 , piece_id=6O5SOAOMFFQCGLAMH6FP6MXCGF7AW3IDCEB7P4XLH3B2PQKDPYAQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.482488858 (500h44m45.353571903s) [noise=true , address=77.90.31.19:4844 , piece_id=W3VTA6OFW2VKMT5OV6NRWF67XSCTGYZYCHNYBT6G4MQLUO5VACRA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.484157225 (500h44m45.35524026s) [noise=true , address=145.79.227.36:4518 , piece_id=GB5EJMA4HEQ3OZRZPTHXIDIJPR2JM2GB2QBFGKSNQFXZ27VYDGJA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.484820698 (500h44m45.355903733s) [noise=true , address=145.79.227.38:4977 , piece_id=LWFE4J6SHUJHGTG4FJCCDLRMDVF63OJIMRIYYXIHRIGN6QOM4GPQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.485855318 (500h44m45.356938364s) [noise=true , address=145.79.227.12:4765 , piece_id=2KSU3DJ4YPMHGTTV4ERPSXT6ESH62BXCDDB2OYIENAE326ISLEEQ, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.486269086 (500h44m45.357352132s) [noise=true , address=45.91.116.6:4975 , piece_id=3LGG5PJD55TKBMWASWWMXWZZU5LYBSVFWMUCUAR4KXFCYHQFMZZA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.487477217 (500h44m45.358560262s) [noise=true , address=45.91.116.12:4226 , piece_id=XVP2IRDGYENVQX3TLZ7RMFKTFU43BO3PIWCDSGQYGJYKGV5RSVQA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:45.488656142 (500h44m45.359739177s) [noise=true , address=145.79.227.16:4910 , piece_id=NTCRLL3UKVE74QPWIXOZDAZS62PU3ALU6GZBLAOU2XUFPVL6PALA, seg_pos={0 0}, upload=439480]: Uploading piece... Dec 26 12:13:47.209954552 (500h44m47.081037597s) [upload=439480]: Next returned nil segment Dec 26 12:13:47.209981624 (500h44m47.081064679s) [upload=439480]: Waiting for error group managing segments... Dec 26 12:13:47.494435664 (500h44m47.365518699s) [noise=true , address=145.79.227.50:4374 , piece_id=NM475EKJFE2BSD34O4FWQHCR4AYAK3FL2ZDZ4ARAIW46SRL3HDFA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.496892482 (500h44m47.367975517s) [noise=true , address=145.79.227.34:4906 , piece_id=YYXTKLUQHUHKVKYDPIP5HA5DI5H6Z5RKD3FFRFUCGZQHQHPARHOQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.500452510 (500h44m47.371535556s) [noise=true , address=145.79.227.3:4150 , piece_id=CO33YMIFPGDM2ATUAPIKPLLKA74CUH4LZFEBNQ3BJLH2C5IMNAKQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.501376009 (500h44m47.372459044s) [noise=true , address=145.79.227.48:4303 , piece_id=6O5SOAOMFFQCGLAMH6FP6MXCGF7AW3IDCEB7P4XLH3B2PQKDPYAQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.502407073 (500h44m47.373490118s) [noise=true , address=145.79.227.45:4230 , piece_id=GPWMKUXOY22V6P3QCFRRUKT7RHAVRXF3GYVI4HA2HA4VK573W32Q, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.507406572 (500h44m47.378489607s) [noise=true , address=145.79.227.4:4185 , piece_id=FCWWRPAMMVUD55JEGOOTOITVCQRP2OLEVQMGQHRALTJE3Z4XZ6EQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.507503537 (500h44m47.378586572s) [noise=true , address=145.79.227.35:4483 , piece_id=LPYLAMXLIUDNNQSFTKQ2OHQZ65XMEQHGJJ26BET45F5QRZZ6KIFA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.510095452 (500h44m47.381178487s) [noise=true , address=145.79.227.38:4977 , piece_id=LWFE4J6SHUJHGTG4FJCCDLRMDVF63OJIMRIYYXIHRIGN6QOM4GPQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.510758976 (500h44m47.381842031s) [noise=true , address=145.79.227.36:4518 , piece_id=GB5EJMA4HEQ3OZRZPTHXIDIJPR2JM2GB2QBFGKSNQFXZ27VYDGJA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.512636902 (500h44m47.383719937s) [noise=true , address=145.79.227.16:4910 , piece_id=NTCRLL3UKVE74QPWIXOZDAZS62PU3ALU6GZBLAOU2XUFPVL6PALA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.517959307 (500h44m47.389042342s) [noise=true , address=145.79.227.12:4765 , piece_id=2KSU3DJ4YPMHGTTV4ERPSXT6ESH62BXCDDB2OYIENAE326ISLEEQ, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.799208164 (500h44m47.670291209s) [noise=true , address=45.91.116.6:4975 , piece_id=3LGG5PJD55TKBMWASWWMXWZZU5LYBSVFWMUCUAR4KXFCYHQFMZZA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.823503727 (500h44m47.694586772s) [noise=true , address=77.90.31.15:4700 , piece_id=XQ7LOV3E3IOYSHVETVEUQIBILQPNYKT2UZYUFZE4FM6DLJ3ADXPA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: <nil> Dec 26 12:13:47.823528834 (500h44m47.694611869s) [seg_pos={0 0}, upload=439480]: Segment reached optimal threshold of 13 pieces. Dec 26 12:13:47.823719397 (500h44m47.694802432s) [noise=true , address=45.91.116.10:4082 , piece_id=VCKFBL4SI7T4IM3RBCXTPBSKRA2KM7NR54RHBSBDFIOE52XYNHGA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 26 12:13:47.823729867 (500h44m47.694812922s) [noise=true , address=45.91.116.10:4082 , piece_id=VCKFBL4SI7T4IM3RBCXTPBSKRA2KM7NR54RHBSBDFIOE52XYNHGA, seg_pos={0 0}, upload=439480]: Long tail cancellation due to optimal threshold Dec 26 12:13:47.823775204 (500h44m47.694858239s) [noise=true , address=77.90.31.19:4844 , piece_id=W3VTA6OFW2VKMT5OV6NRWF67XSCTGYZYCHNYBT6G4MQLUO5VACRA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:43582->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:43582->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43582->77.90.31.19:4844: use of closed network connection Dec 26 12:13:47.823783219 (500h44m47.694866254s) [noise=true , address=77.90.31.19:4844 , piece_id=W3VTA6OFW2VKMT5OV6NRWF67XSCTGYZYCHNYBT6G4MQLUO5VACRA, seg_pos={0 0}, upload=439480]: Long tail cancellation due to optimal threshold Dec 26 12:13:47.823964143 (500h44m47.695047198s) [noise=true , address=45.91.116.12:4226 , piece_id=XVP2IRDGYENVQX3TLZ7RMFKTFU43BO3PIWCDSGQYGJYKGV5RSVQA, seg_pos={0 0}, upload=439480]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 26 12:13:47.823973531 (500h44m47.695056576s) [noise=true , address=45.91.116.12:4226 , piece_id=XVP2IRDGYENVQX3TLZ7RMFKTFU43BO3PIWCDSGQYGJYKGV5RSVQA, seg_pos={0 0}, upload=439480]: Long tail cancellation due to optimal threshold Dec 26 12:13:47.823995603 (500h44m47.695078648s) [seg_pos={0 0}, upload=439480]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:13:47.824073261 (500h44m47.695156296s) [upload=439480]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:13:47.824079472 (500h44m47.695162507s) [upload=439480]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:13:48.108768200 (500h44m47.979851245s) [upload=439480]: Done uploading segments... Dec 26 12:13:48.108793547 (500h44m47.979876592s) [upload=439480]: Upload finished. err: <nil> Duration: 30.004169596s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.374878445s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 12:13:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.374908292s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....................X....X............XX........X..X......X.] duration: 3.497546084s trace: https://jaeger.dev.storj.io/trace/23358163356fbf1a FAIL (2025-12-26 12:13:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497584366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.354359946s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 12:13:13 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.354385906s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X.X...X....X......X.X.X...X.X........X..XXXXX...X.X....X..] duration: 5.936471008s trace: https://jaeger.dev.storj.io/trace/2441c7817cf7fe81 FAIL (2025-12-26 12:12:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27111 (500h42m50.521210146s) [upload=439454]: Done starting upload Dec 26 12:11:50.650224937 (500h42m50.521307992s) [upload=439454]: Uploading segments... Dec 26 12:11:50.650394280 (500h42m50.521477335s) [upload=439454]: Got next segment. Inline: false Dec 26 12:11:50.650432552 (500h42m50.521515598s) [upload=439454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:11:50.650438985 (500h42m50.52152203s) [upload=439454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:11:50.896787454 (500h42m50.767870489s) [seg_pos={0 0}, upload=439454]: Begin upload segment... Dec 26 12:11:50.896844142 (500h42m50.767927187s) [seg_pos={0 0}, upload=439454]: Done begin upload segment. Dec 26 12:11:50.896864912 (500h42m50.767947957s) [noise=true , address=45.91.116.10:4084 , piece_id=PVLIO4NZLTEB2ZUFBPA2BH3IX7WWQKN5OWIRLFVV4U4QVMT3OJMA, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.896879870 (500h42m50.767962915s) [noise=true , address=145.79.227.6:4258 , piece_id=FOYYJNK3CVM4LV5KGLC5LPDUDK64WTGWPFEOG3KMG2ILXLTUSO2A, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.897463502 (500h42m50.768546547s) [noise=true , address=77.90.31.18:4808 , piece_id=TA53HTSQ74AVHOIXDLNRISONYSE3ELASRAWATFX63VCA2V4QDDQQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.897693249 (500h42m50.768776294s) [noise=true , address=145.79.227.3:4150 , piece_id=ABVNWCLGA6UT2NLXGSCM2352DWHFNIIJBGGGWENS2UX7L4GBRTAQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.898106896 (500h42m50.769189941s) [noise=true , address=77.90.31.14:4663 , piece_id=I5BSW2OBLAJTYWHNF42ELCOXW4Z3VJMBPITN7XFGSZKVIUB526EA, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.899620509 (500h42m50.770703554s) [noise=true , address=145.79.227.12:4765 , piece_id=YHBP3QQUUTHFXJ3UQ2X23OMZ4KADSLSER3UP3SZW3QDWVHYNII5A, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.899929217 (500h42m50.771012252s) [noise=true , address=145.79.227.44:4193 , piece_id=7QDHFMOKUAIAAGPIPVVXFC7WHBXCN22COIKPZAKWUPRVQZAQCU4Q, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.900206635 (500h42m50.77128968s) [noise=true , address=145.79.227.11:4731 , piece_id=N5TQGW5VXD55ABQMDNU5UOMH5XQZ44VHCL5KCVMKX6VYXTFGUS2Q, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.900517166 (500h42m50.771600201s) [noise=true , address=145.79.227.15:4874 , piece_id=VZPRJT3JC7T3UKD3QJMQGEB54WDGYZP6MMGBFBMTTPF5IPQNCTFQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.900943358 (500h42m50.772026403s) [noise=true , address=45.91.116.5:4297 , piece_id=VVOWLC56Y6YZXIFKMHQMSTYSIEB7T2LCWL2NNL3BRWE4WQWESATQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.901344581 (500h42m50.772427627s) [noise=true , address=45.91.116.6:4974 , piece_id=G45PD4LEBEKAURU7XRXUVBKPB225BMXTVY2B3HKQXRZ2AECMC7RA, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.901687204 (500h42m50.772770239s) [noise=true , address=145.79.227.50:4374 , piece_id=CFLM4BHSRVK7S762KEVXEMIHIQTPWTZ6NC525GS7FWPCIIUGXYCQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.902063320 (500h42m50.773146355s) [noise=true , address=145.79.227.22:4511 , piece_id=UUONQ37CWCOHPCQDL3PQLKPAFVYB3IQBJKXSQ6QJARN4VBL3NJ4Q, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.903260570 (500h42m50.774343605s) [noise=true , address=77.90.31.17:4773 , piece_id=SKB3OKNQIJBWVE5FMDYWLQG7L2D6LBCIKZAS6UJXNQ2MCWII4SNQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.904425098 (500h42m50.775508133s) [noise=true , address=145.79.227.16:4912 , piece_id=4MKWS3QMUW2TEQZEEWJZONGKCAGALJUO2QSWXFFFHQYULUFUBVIQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.905516055 (500h42m50.7765991s) [noise=true , address=145.79.227.40:4051 , piece_id=6MODZBHXN5RS4US4UGAEIXJJ2VOMT7VAINEJMX7H7IPUGG3EPAFQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:52.623144428 (500h42m52.494227473s) [upload=439454]: Next returned nil segment Dec 26 12:11:52.623225853 (500h42m52.494308888s) [upload=439454]: Waiting for error group managing segments... Dec 26 12:11:52.911692243 (500h42m52.782775278s) [noise=true , address=145.79.227.16:4912 , piece_id=4MKWS3QMUW2TEQZEEWJZONGKCAGALJUO2QSWXFFFHQYULUFUBVIQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.914322070 (500h42m52.785405105s) [noise=true , address=145.79.227.22:4511 , piece_id=UUONQ37CWCOHPCQDL3PQLKPAFVYB3IQBJKXSQ6QJARN4VBL3NJ4Q, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.931833955 (500h42m52.80291699s) [noise=true , address=145.79.227.50:4374 , piece_id=CFLM4BHSRVK7S762KEVXEMIHIQTPWTZ6NC525GS7FWPCIIUGXYCQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.934427695 (500h42m52.80551073s) [noise=true , address=145.79.227.11:4731 , piece_id=N5TQGW5VXD55ABQMDNU5UOMH5XQZ44VHCL5KCVMKX6VYXTFGUS2Q, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.934782410 (500h42m52.805865445s) [noise=true , address=145.79.227.3:4150 , piece_id=ABVNWCLGA6UT2NLXGSCM2352DWHFNIIJBGGGWENS2UX7L4GBRTAQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.934855138 (500h42m52.805938173s) [noise=true , address=145.79.227.12:4765 , piece_id=YHBP3QQUUTHFXJ3UQ2X23OMZ4KADSLSER3UP3SZW3QDWVHYNII5A, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.935130322 (500h42m52.806213368s) [noise=true , address=145.79.227.6:4258 , piece_id=FOYYJNK3CVM4LV5KGLC5LPDUDK64WTGWPFEOG3KMG2ILXLTUSO2A, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.939770328 (500h42m52.810853363s) [noise=true , address=145.79.227.15:4874 , piece_id=VZPRJT3JC7T3UKD3QJMQGEB54WDGYZP6MMGBFBMTTPF5IPQNCTFQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.962925690 (500h42m52.834008725s) [noise=true , address=145.79.227.40:4051 , piece_id=6MODZBHXN5RS4US4UGAEIXJJ2VOMT7VAINEJMX7H7IPUGG3EPAFQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.081843280 (500h42m52.952926325s) [noise=true , address=77.90.31.18:4808 , piece_id=TA53HTSQ74AVHOIXDLNRISONYSE3ELASRAWATFX63VCA2V4QDDQQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.157134046 (500h42m53.028217092s) [noise=true , address=45.91.116.6:4974 , piece_id=G45PD4LEBEKAURU7XRXUVBKPB225BMXTVY2B3HKQXRZ2AECMC7RA, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.277575808 (500h42m53.148658844s) [noise=true , address=45.91.116.10:4084 , piece_id=PVLIO4NZLTEB2ZUFBPA2BH3IX7WWQKN5OWIRLFVV4U4QVMT3OJMA, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.487886897 (500h42m53.358969932s) [noise=true , address=77.90.31.14:4663 , piece_id=I5BSW2OBLAJTYWHNF42ELCOXW4Z3VJMBPITN7XFGSZKVIUB526EA, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.487906675 (500h42m53.35898971s) [seg_pos={0 0}, upload=439454]: Segment reached optimal threshold of 13 pieces. Dec 26 12:11:53.488240670 (500h42m53.359323705s) [noise=true , address=145.79.227.44:4193 , piece_id=7QDHFMOKUAIAAGPIPVVXFC7WHBXCN22COIKPZAKWUPRVQZAQCU4Q, seg_pos={0 0}, upload=439454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 12:11:53.488247814 (500h42m53.359330849s) [noise=true , address=145.79.227.44:4193 , piece_id=7QDHFMOKUAIAAGPIPVVXFC7WHBXCN22COIKPZAKWUPRVQZAQCU4Q, seg_pos={0 0}, upload=439454]: Long tail cancellation due to optimal threshold Dec 26 12:11:53.488369746 (500h42m53.359452781s) [noise=true , address=45.91.116.5:4297 , piece_id=VVOWLC56Y6YZXIFKMHQMSTYSIEB7T2LCWL2NNL3BRWE4WQWESATQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:42324->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:42324->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42324->45.91.116.5:4297: use of closed network connection Dec 26 12:11:53.488376288 (500h42m53.359459333s) [noise=true , address=45.91.116.5:4297 , piece_id=VVOWLC56Y6YZXIFKMHQMSTYSIEB7T2LCWL2NNL3BRWE4WQWESATQ, seg_pos={0 0}, upload=439454]: Long tail cancellation due to optimal threshold Dec 26 12:11:53.488532737 (500h42m53.359615782s) [noise=true , address=77.90.31.17:4773 , piece_id=SKB3OKNQIJBWVE5FMDYWLQG7L2D6LBCIKZAS6UJXNQ2MCWII4SNQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: read tcp 188.166.225.170:46630->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:11:53.488538427 (500h42m53.359621462s) [noise=true , address=77.90.31.17:4773 , piece_id=SKB3OKNQIJBWVE5FMDYWLQG7L2D6LBCIKZAS6UJXNQ2MCWII4SNQ, seg_pos={0 0}, upload=439454]: Long tail cancellation due to optimal threshold Dec 26 12:11:53.488551813 (500h42m53.359634848s) [seg_pos={0 0}, upload=439454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:11:53.488648136 (500h42m53.359731171s) [upload=439454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:11:53.488652554 (500h42m53.35973559s) [upload=439454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:11:53.786818875 (500h42m53.65790191s) [upload=439454]: Done uploading segments... Dec 26 12:11:53.786838763 (500h42m53.657921798s) [upload=439454]: Upload finished. err: <nil> Duration: 5.936514401s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.423048902s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 12:12:41 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.423078719s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX.X.X...X....X......X.X.X...X.X........X..XXXXX...X.X....X.] duration: 5.96405273s trace: https://jaeger.dev.storj.io/trace/2441c7817cf7fe81 FAIL (2025-12-26 12:12:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27111 (500h42m50.521210146s) [upload=439454]: Done starting upload Dec 26 12:11:50.650224937 (500h42m50.521307992s) [upload=439454]: Uploading segments... Dec 26 12:11:50.650394280 (500h42m50.521477335s) [upload=439454]: Got next segment. Inline: false Dec 26 12:11:50.650432552 (500h42m50.521515598s) [upload=439454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:11:50.650438985 (500h42m50.52152203s) [upload=439454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:11:50.896787454 (500h42m50.767870489s) [seg_pos={0 0}, upload=439454]: Begin upload segment... Dec 26 12:11:50.896844142 (500h42m50.767927187s) [seg_pos={0 0}, upload=439454]: Done begin upload segment. Dec 26 12:11:50.896864912 (500h42m50.767947957s) [noise=true , address=45.91.116.10:4084 , piece_id=PVLIO4NZLTEB2ZUFBPA2BH3IX7WWQKN5OWIRLFVV4U4QVMT3OJMA, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.896879870 (500h42m50.767962915s) [noise=true , address=145.79.227.6:4258 , piece_id=FOYYJNK3CVM4LV5KGLC5LPDUDK64WTGWPFEOG3KMG2ILXLTUSO2A, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.897463502 (500h42m50.768546547s) [noise=true , address=77.90.31.18:4808 , piece_id=TA53HTSQ74AVHOIXDLNRISONYSE3ELASRAWATFX63VCA2V4QDDQQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.897693249 (500h42m50.768776294s) [noise=true , address=145.79.227.3:4150 , piece_id=ABVNWCLGA6UT2NLXGSCM2352DWHFNIIJBGGGWENS2UX7L4GBRTAQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.898106896 (500h42m50.769189941s) [noise=true , address=77.90.31.14:4663 , piece_id=I5BSW2OBLAJTYWHNF42ELCOXW4Z3VJMBPITN7XFGSZKVIUB526EA, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.899620509 (500h42m50.770703554s) [noise=true , address=145.79.227.12:4765 , piece_id=YHBP3QQUUTHFXJ3UQ2X23OMZ4KADSLSER3UP3SZW3QDWVHYNII5A, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.899929217 (500h42m50.771012252s) [noise=true , address=145.79.227.44:4193 , piece_id=7QDHFMOKUAIAAGPIPVVXFC7WHBXCN22COIKPZAKWUPRVQZAQCU4Q, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.900206635 (500h42m50.77128968s) [noise=true , address=145.79.227.11:4731 , piece_id=N5TQGW5VXD55ABQMDNU5UOMH5XQZ44VHCL5KCVMKX6VYXTFGUS2Q, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.900517166 (500h42m50.771600201s) [noise=true , address=145.79.227.15:4874 , piece_id=VZPRJT3JC7T3UKD3QJMQGEB54WDGYZP6MMGBFBMTTPF5IPQNCTFQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.900943358 (500h42m50.772026403s) [noise=true , address=45.91.116.5:4297 , piece_id=VVOWLC56Y6YZXIFKMHQMSTYSIEB7T2LCWL2NNL3BRWE4WQWESATQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.901344581 (500h42m50.772427627s) [noise=true , address=45.91.116.6:4974 , piece_id=G45PD4LEBEKAURU7XRXUVBKPB225BMXTVY2B3HKQXRZ2AECMC7RA, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.901687204 (500h42m50.772770239s) [noise=true , address=145.79.227.50:4374 , piece_id=CFLM4BHSRVK7S762KEVXEMIHIQTPWTZ6NC525GS7FWPCIIUGXYCQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.902063320 (500h42m50.773146355s) [noise=true , address=145.79.227.22:4511 , piece_id=UUONQ37CWCOHPCQDL3PQLKPAFVYB3IQBJKXSQ6QJARN4VBL3NJ4Q, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.903260570 (500h42m50.774343605s) [noise=true , address=77.90.31.17:4773 , piece_id=SKB3OKNQIJBWVE5FMDYWLQG7L2D6LBCIKZAS6UJXNQ2MCWII4SNQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.904425098 (500h42m50.775508133s) [noise=true , address=145.79.227.16:4912 , piece_id=4MKWS3QMUW2TEQZEEWJZONGKCAGALJUO2QSWXFFFHQYULUFUBVIQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.905516055 (500h42m50.7765991s) [noise=true , address=145.79.227.40:4051 , piece_id=6MODZBHXN5RS4US4UGAEIXJJ2VOMT7VAINEJMX7H7IPUGG3EPAFQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:52.623144428 (500h42m52.494227473s) [upload=439454]: Next returned nil segment Dec 26 12:11:52.623225853 (500h42m52.494308888s) [upload=439454]: Waiting for error group managing segments... Dec 26 12:11:52.911692243 (500h42m52.782775278s) [noise=true , address=145.79.227.16:4912 , piece_id=4MKWS3QMUW2TEQZEEWJZONGKCAGALJUO2QSWXFFFHQYULUFUBVIQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.914322070 (500h42m52.785405105s) [noise=true , address=145.79.227.22:4511 , piece_id=UUONQ37CWCOHPCQDL3PQLKPAFVYB3IQBJKXSQ6QJARN4VBL3NJ4Q, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.931833955 (500h42m52.80291699s) [noise=true , address=145.79.227.50:4374 , piece_id=CFLM4BHSRVK7S762KEVXEMIHIQTPWTZ6NC525GS7FWPCIIUGXYCQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.934427695 (500h42m52.80551073s) [noise=true , address=145.79.227.11:4731 , piece_id=N5TQGW5VXD55ABQMDNU5UOMH5XQZ44VHCL5KCVMKX6VYXTFGUS2Q, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.934782410 (500h42m52.805865445s) [noise=true , address=145.79.227.3:4150 , piece_id=ABVNWCLGA6UT2NLXGSCM2352DWHFNIIJBGGGWENS2UX7L4GBRTAQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.934855138 (500h42m52.805938173s) [noise=true , address=145.79.227.12:4765 , piece_id=YHBP3QQUUTHFXJ3UQ2X23OMZ4KADSLSER3UP3SZW3QDWVHYNII5A, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.935130322 (500h42m52.806213368s) [noise=true , address=145.79.227.6:4258 , piece_id=FOYYJNK3CVM4LV5KGLC5LPDUDK64WTGWPFEOG3KMG2ILXLTUSO2A, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.939770328 (500h42m52.810853363s) [noise=true , address=145.79.227.15:4874 , piece_id=VZPRJT3JC7T3UKD3QJMQGEB54WDGYZP6MMGBFBMTTPF5IPQNCTFQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.962925690 (500h42m52.834008725s) [noise=true , address=145.79.227.40:4051 , piece_id=6MODZBHXN5RS4US4UGAEIXJJ2VOMT7VAINEJMX7H7IPUGG3EPAFQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.081843280 (500h42m52.952926325s) [noise=true , address=77.90.31.18:4808 , piece_id=TA53HTSQ74AVHOIXDLNRISONYSE3ELASRAWATFX63VCA2V4QDDQQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.157134046 (500h42m53.028217092s) [noise=true , address=45.91.116.6:4974 , piece_id=G45PD4LEBEKAURU7XRXUVBKPB225BMXTVY2B3HKQXRZ2AECMC7RA, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.277575808 (500h42m53.148658844s) [noise=true , address=45.91.116.10:4084 , piece_id=PVLIO4NZLTEB2ZUFBPA2BH3IX7WWQKN5OWIRLFVV4U4QVMT3OJMA, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.487886897 (500h42m53.358969932s) [noise=true , address=77.90.31.14:4663 , piece_id=I5BSW2OBLAJTYWHNF42ELCOXW4Z3VJMBPITN7XFGSZKVIUB526EA, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.487906675 (500h42m53.35898971s) [seg_pos={0 0}, upload=439454]: Segment reached optimal threshold of 13 pieces. Dec 26 12:11:53.488240670 (500h42m53.359323705s) [noise=true , address=145.79.227.44:4193 , piece_id=7QDHFMOKUAIAAGPIPVVXFC7WHBXCN22COIKPZAKWUPRVQZAQCU4Q, seg_pos={0 0}, upload=439454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 12:11:53.488247814 (500h42m53.359330849s) [noise=true , address=145.79.227.44:4193 , piece_id=7QDHFMOKUAIAAGPIPVVXFC7WHBXCN22COIKPZAKWUPRVQZAQCU4Q, seg_pos={0 0}, upload=439454]: Long tail cancellation due to optimal threshold Dec 26 12:11:53.488369746 (500h42m53.359452781s) [noise=true , address=45.91.116.5:4297 , piece_id=VVOWLC56Y6YZXIFKMHQMSTYSIEB7T2LCWL2NNL3BRWE4WQWESATQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:42324->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:42324->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42324->45.91.116.5:4297: use of closed network connection Dec 26 12:11:53.488376288 (500h42m53.359459333s) [noise=true , address=45.91.116.5:4297 , piece_id=VVOWLC56Y6YZXIFKMHQMSTYSIEB7T2LCWL2NNL3BRWE4WQWESATQ, seg_pos={0 0}, upload=439454]: Long tail cancellation due to optimal threshold Dec 26 12:11:53.488532737 (500h42m53.359615782s) [noise=true , address=77.90.31.17:4773 , piece_id=SKB3OKNQIJBWVE5FMDYWLQG7L2D6LBCIKZAS6UJXNQ2MCWII4SNQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: read tcp 188.166.225.170:46630->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:11:53.488538427 (500h42m53.359621462s) [noise=true , address=77.90.31.17:4773 , piece_id=SKB3OKNQIJBWVE5FMDYWLQG7L2D6LBCIKZAS6UJXNQ2MCWII4SNQ, seg_pos={0 0}, upload=439454]: Long tail cancellation due to optimal threshold Dec 26 12:11:53.488551813 (500h42m53.359634848s) [seg_pos={0 0}, upload=439454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:11:53.488648136 (500h42m53.359731171s) [upload=439454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:11:53.488652554 (500h42m53.35973559s) [upload=439454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:11:53.786818875 (500h42m53.65790191s) [upload=439454]: Done uploading segments... Dec 26 12:11:53.786838763 (500h42m53.657921798s) [upload=439454]: Upload finished. err: <nil> Duration: 5.964073149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX.X.X...X....X......X.X.X...X.X........X..XXXXX...X.X....X] duration: 30.004620336s trace: https://jaeger.dev.storj.io/trace/2441c7817cf7fe81 FAIL (2025-12-26 12:12:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27111 (500h42m50.521210146s) [upload=439454]: Done starting upload Dec 26 12:11:50.650224937 (500h42m50.521307992s) [upload=439454]: Uploading segments... Dec 26 12:11:50.650394280 (500h42m50.521477335s) [upload=439454]: Got next segment. Inline: false Dec 26 12:11:50.650432552 (500h42m50.521515598s) [upload=439454]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:11:50.650438985 (500h42m50.52152203s) [upload=439454]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:11:50.896787454 (500h42m50.767870489s) [seg_pos={0 0}, upload=439454]: Begin upload segment... Dec 26 12:11:50.896844142 (500h42m50.767927187s) [seg_pos={0 0}, upload=439454]: Done begin upload segment. Dec 26 12:11:50.896864912 (500h42m50.767947957s) [noise=true , address=45.91.116.10:4084 , piece_id=PVLIO4NZLTEB2ZUFBPA2BH3IX7WWQKN5OWIRLFVV4U4QVMT3OJMA, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.896879870 (500h42m50.767962915s) [noise=true , address=145.79.227.6:4258 , piece_id=FOYYJNK3CVM4LV5KGLC5LPDUDK64WTGWPFEOG3KMG2ILXLTUSO2A, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.897463502 (500h42m50.768546547s) [noise=true , address=77.90.31.18:4808 , piece_id=TA53HTSQ74AVHOIXDLNRISONYSE3ELASRAWATFX63VCA2V4QDDQQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.897693249 (500h42m50.768776294s) [noise=true , address=145.79.227.3:4150 , piece_id=ABVNWCLGA6UT2NLXGSCM2352DWHFNIIJBGGGWENS2UX7L4GBRTAQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.898106896 (500h42m50.769189941s) [noise=true , address=77.90.31.14:4663 , piece_id=I5BSW2OBLAJTYWHNF42ELCOXW4Z3VJMBPITN7XFGSZKVIUB526EA, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.899620509 (500h42m50.770703554s) [noise=true , address=145.79.227.12:4765 , piece_id=YHBP3QQUUTHFXJ3UQ2X23OMZ4KADSLSER3UP3SZW3QDWVHYNII5A, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.899929217 (500h42m50.771012252s) [noise=true , address=145.79.227.44:4193 , piece_id=7QDHFMOKUAIAAGPIPVVXFC7WHBXCN22COIKPZAKWUPRVQZAQCU4Q, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.900206635 (500h42m50.77128968s) [noise=true , address=145.79.227.11:4731 , piece_id=N5TQGW5VXD55ABQMDNU5UOMH5XQZ44VHCL5KCVMKX6VYXTFGUS2Q, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.900517166 (500h42m50.771600201s) [noise=true , address=145.79.227.15:4874 , piece_id=VZPRJT3JC7T3UKD3QJMQGEB54WDGYZP6MMGBFBMTTPF5IPQNCTFQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.900943358 (500h42m50.772026403s) [noise=true , address=45.91.116.5:4297 , piece_id=VVOWLC56Y6YZXIFKMHQMSTYSIEB7T2LCWL2NNL3BRWE4WQWESATQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.901344581 (500h42m50.772427627s) [noise=true , address=45.91.116.6:4974 , piece_id=G45PD4LEBEKAURU7XRXUVBKPB225BMXTVY2B3HKQXRZ2AECMC7RA, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.901687204 (500h42m50.772770239s) [noise=true , address=145.79.227.50:4374 , piece_id=CFLM4BHSRVK7S762KEVXEMIHIQTPWTZ6NC525GS7FWPCIIUGXYCQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.902063320 (500h42m50.773146355s) [noise=true , address=145.79.227.22:4511 , piece_id=UUONQ37CWCOHPCQDL3PQLKPAFVYB3IQBJKXSQ6QJARN4VBL3NJ4Q, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.903260570 (500h42m50.774343605s) [noise=true , address=77.90.31.17:4773 , piece_id=SKB3OKNQIJBWVE5FMDYWLQG7L2D6LBCIKZAS6UJXNQ2MCWII4SNQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.904425098 (500h42m50.775508133s) [noise=true , address=145.79.227.16:4912 , piece_id=4MKWS3QMUW2TEQZEEWJZONGKCAGALJUO2QSWXFFFHQYULUFUBVIQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:50.905516055 (500h42m50.7765991s) [noise=true , address=145.79.227.40:4051 , piece_id=6MODZBHXN5RS4US4UGAEIXJJ2VOMT7VAINEJMX7H7IPUGG3EPAFQ, seg_pos={0 0}, upload=439454]: Uploading piece... Dec 26 12:11:52.623144428 (500h42m52.494227473s) [upload=439454]: Next returned nil segment Dec 26 12:11:52.623225853 (500h42m52.494308888s) [upload=439454]: Waiting for error group managing segments... Dec 26 12:11:52.911692243 (500h42m52.782775278s) [noise=true , address=145.79.227.16:4912 , piece_id=4MKWS3QMUW2TEQZEEWJZONGKCAGALJUO2QSWXFFFHQYULUFUBVIQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.914322070 (500h42m52.785405105s) [noise=true , address=145.79.227.22:4511 , piece_id=UUONQ37CWCOHPCQDL3PQLKPAFVYB3IQBJKXSQ6QJARN4VBL3NJ4Q, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.931833955 (500h42m52.80291699s) [noise=true , address=145.79.227.50:4374 , piece_id=CFLM4BHSRVK7S762KEVXEMIHIQTPWTZ6NC525GS7FWPCIIUGXYCQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.934427695 (500h42m52.80551073s) [noise=true , address=145.79.227.11:4731 , piece_id=N5TQGW5VXD55ABQMDNU5UOMH5XQZ44VHCL5KCVMKX6VYXTFGUS2Q, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.934782410 (500h42m52.805865445s) [noise=true , address=145.79.227.3:4150 , piece_id=ABVNWCLGA6UT2NLXGSCM2352DWHFNIIJBGGGWENS2UX7L4GBRTAQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.934855138 (500h42m52.805938173s) [noise=true , address=145.79.227.12:4765 , piece_id=YHBP3QQUUTHFXJ3UQ2X23OMZ4KADSLSER3UP3SZW3QDWVHYNII5A, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.935130322 (500h42m52.806213368s) [noise=true , address=145.79.227.6:4258 , piece_id=FOYYJNK3CVM4LV5KGLC5LPDUDK64WTGWPFEOG3KMG2ILXLTUSO2A, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.939770328 (500h42m52.810853363s) [noise=true , address=145.79.227.15:4874 , piece_id=VZPRJT3JC7T3UKD3QJMQGEB54WDGYZP6MMGBFBMTTPF5IPQNCTFQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:52.962925690 (500h42m52.834008725s) [noise=true , address=145.79.227.40:4051 , piece_id=6MODZBHXN5RS4US4UGAEIXJJ2VOMT7VAINEJMX7H7IPUGG3EPAFQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.081843280 (500h42m52.952926325s) [noise=true , address=77.90.31.18:4808 , piece_id=TA53HTSQ74AVHOIXDLNRISONYSE3ELASRAWATFX63VCA2V4QDDQQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.157134046 (500h42m53.028217092s) [noise=true , address=45.91.116.6:4974 , piece_id=G45PD4LEBEKAURU7XRXUVBKPB225BMXTVY2B3HKQXRZ2AECMC7RA, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.277575808 (500h42m53.148658844s) [noise=true , address=45.91.116.10:4084 , piece_id=PVLIO4NZLTEB2ZUFBPA2BH3IX7WWQKN5OWIRLFVV4U4QVMT3OJMA, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.487886897 (500h42m53.358969932s) [noise=true , address=77.90.31.14:4663 , piece_id=I5BSW2OBLAJTYWHNF42ELCOXW4Z3VJMBPITN7XFGSZKVIUB526EA, seg_pos={0 0}, upload=439454]: Done uploading piece. err: <nil> Dec 26 12:11:53.487906675 (500h42m53.35898971s) [seg_pos={0 0}, upload=439454]: Segment reached optimal threshold of 13 pieces. Dec 26 12:11:53.488240670 (500h42m53.359323705s) [noise=true , address=145.79.227.44:4193 , piece_id=7QDHFMOKUAIAAGPIPVVXFC7WHBXCN22COIKPZAKWUPRVQZAQCU4Q, seg_pos={0 0}, upload=439454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 12:11:53.488247814 (500h42m53.359330849s) [noise=true , address=145.79.227.44:4193 , piece_id=7QDHFMOKUAIAAGPIPVVXFC7WHBXCN22COIKPZAKWUPRVQZAQCU4Q, seg_pos={0 0}, upload=439454]: Long tail cancellation due to optimal threshold Dec 26 12:11:53.488369746 (500h42m53.359452781s) [noise=true , address=45.91.116.5:4297 , piece_id=VVOWLC56Y6YZXIFKMHQMSTYSIEB7T2LCWL2NNL3BRWE4WQWESATQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:42324->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:42324->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42324->45.91.116.5:4297: use of closed network connection Dec 26 12:11:53.488376288 (500h42m53.359459333s) [noise=true , address=45.91.116.5:4297 , piece_id=VVOWLC56Y6YZXIFKMHQMSTYSIEB7T2LCWL2NNL3BRWE4WQWESATQ, seg_pos={0 0}, upload=439454]: Long tail cancellation due to optimal threshold Dec 26 12:11:53.488532737 (500h42m53.359615782s) [noise=true , address=77.90.31.17:4773 , piece_id=SKB3OKNQIJBWVE5FMDYWLQG7L2D6LBCIKZAS6UJXNQ2MCWII4SNQ, seg_pos={0 0}, upload=439454]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: read tcp 188.166.225.170:46630->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:11:53.488538427 (500h42m53.359621462s) [noise=true , address=77.90.31.17:4773 , piece_id=SKB3OKNQIJBWVE5FMDYWLQG7L2D6LBCIKZAS6UJXNQ2MCWII4SNQ, seg_pos={0 0}, upload=439454]: Long tail cancellation due to optimal threshold Dec 26 12:11:53.488551813 (500h42m53.359634848s) [seg_pos={0 0}, upload=439454]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:11:53.488648136 (500h42m53.359731171s) [upload=439454]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:11:53.488652554 (500h42m53.35973559s) [upload=439454]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:11:53.786818875 (500h42m53.65790191s) [upload=439454]: Done uploading segments... Dec 26 12:11:53.786838763 (500h42m53.657921798s) [upload=439454]: Upload finished. err: <nil> Duration: 30.004644831s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....................X....X............XX........X..X......X] duration: 5.001728858s trace: https://jaeger.dev.storj.io/trace/23358163356fbf1a FAIL (2025-12-26 12:12:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001758965s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.382492454s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 12:11:58 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.382526448s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)