Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.................................X..X..X.X....XX.X.X.X.XX.X.] duration: 6.295721646s trace: https://jaeger.dev.storj.io/trace/6752deb4cacb681b FAIL (2025-12-26 04:42:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: A74HNRTT2GZS2IUELQGOI3WGXZEOPHDGQFQWIFB53D7XMTFIUWSQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 6TUUBT2BEYSINKNNEV7URLMSMSRLPXRYNLX6QDI42JS2PC6YP4AQ) context deadline exceeded ======LOGS====== 04:41:29.493415615 (493h12m29.36449865s) [upload=432886]: Starting upload Dec 26 04:41:29.493480528 (493h12m29.364563563s) [upload=432886]: Done starting upload Dec 26 04:41:29.493610946 (493h12m29.364693982s) [upload=432886]: Uploading segments... Dec 26 04:41:29.493715966 (493h12m29.364799002s) [upload=432886]: Got next segment. Inline: false Dec 26 04:41:29.493759900 (493h12m29.364842975s) [upload=432886]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:41:29.493766834 (493h12m29.364849879s) [upload=432886]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:41:29.734693269 (493h12m29.605776314s) [seg_pos={0 0}, upload=432886]: Begin upload segment... Dec 26 04:41:29.734747452 (493h12m29.605830497s) [seg_pos={0 0}, upload=432886]: Done begin upload segment. Dec 26 04:41:29.734803579 (493h12m29.605886614s) [noise=true , address=45.91.116.12:4226 , piece_id=Y2H2AQVG4DPT7AHYV4OUXJQF3RS55AEVA72HQ4PZOWW2A4MXBOLA, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.734816253 (493h12m29.605899288s) [noise=true , address=145.79.227.14:4837 , piece_id=JDDVPYT7EQI2P76KPRZOPRJUUUTW2VRJ6KMPJGAWI7MV2BO3NKPA, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.735188372 (493h12m29.606271407s) [noise=true , address=45.91.116.14:4189 , piece_id=G7UMR5C5XDWPQ3IAURKZBUEFZ3QXIOL4BDHKZJCEM56OAJAG6PQQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.735456543 (493h12m29.606539578s) [noise=true , address=145.79.227.35:4481 , piece_id=SDCEQ6ZUPWOQWZAXDARIFSCE2E523JAACJI3V4B2HIVK4MVTJ5GA, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.735759158 (493h12m29.606842193s) [noise=true , address=45.91.116.5:4300 , piece_id=6TUUBT2BEYSINKNNEV7URLMSMSRLPXRYNLX6QDI42JS2PC6YP4AQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.736049692 (493h12m29.607132737s) [noise=true , address=45.91.116.8:4047 , piece_id=FM4QI4AOMK4OYJHYHZFCJ6DNZ7X5LOZWVKT4M3QVQMOYA2OGBPVQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.737215782 (493h12m29.608298847s) [noise=true , address=45.91.116.10:4082 , piece_id=A74HNRTT2GZS2IUELQGOI3WGXZEOPHDGQFQWIFB53D7XMTFIUWSQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.737697079 (493h12m29.608780124s) [noise=true , address=145.79.227.52:4446 , piece_id=V76JR4LXHHQQYI4QIN64STDQ2O6S6535UONVQFY6HC4R3FHSAGDQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.738902164 (493h12m29.609985199s) [noise=true , address=145.79.227.45:4231 , piece_id=S6OSKDK3JHMMPY4SIBDYUOIJ3RYRDEJWLA2EHK5HYNVOYHIN35OQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.740154719 (493h12m29.611237754s) [noise=true , address=45.91.116.4:4937 , piece_id=JBURMCQXIOSR5CZ3G6KTQGTENUKBCCDJY5WEYYF6KNJO3XQIS3JQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.741360554 (493h12m29.6124436s) [noise=true , address=145.79.227.15:4876 , piece_id=A6JLSZFAL6BWJMGYF2P4ED4BKPSJVTZRP4RTTZJ5Q773VKU7RMGA, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.743538121 (493h12m29.614621166s) [noise=true , address=145.79.227.38:4979 , piece_id=PQJOGLBHDAHRLJLX7S7EFCJKYCRNW6OUXASOGNP2TRAUUSECLXJA, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.745592874 (493h12m29.616675919s) [noise=true , address=145.79.227.11:4731 , piece_id=V2KTGIIB6AMNPXGIC42Y7SZSXUIT4RQ4OAU46KKTNPUWYPQ4QCSA, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.745716649 (493h12m29.616799684s) [noise=true , address=145.79.227.36:4519 , piece_id=CQUS7OVDXTCNBJOBGRANP6OI5Z575V5LIYWL7ITC33BASV3NVBNQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.747222337 (493h12m29.618305372s) [noise=true , address=145.79.227.48:4302 , piece_id=S3NG7NAHEXKU46IXBMD63RQKCTRILGOZECX2XD2A3ROA2PBIGYGQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.747889948 (493h12m29.618973003s) [noise=true , address=45.91.116.6:4973 , piece_id=NHCX3W2VS6OXRJ737XJNDUF5ZIXTKPX34M7T42OMTGDDBA2BV4FQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:31.347530431 (493h12m31.218613466s) [upload=432886]: Next returned nil segment Dec 26 04:41:31.347550729 (493h12m31.218633764s) [upload=432886]: Waiting for error group managing segments... Dec 26 04:41:31.732161215 (493h12m31.60324425s) [noise=true , address=145.79.227.45:4231 , piece_id=S6OSKDK3JHMMPY4SIBDYUOIJ3RYRDEJWLA2EHK5HYNVOYHIN35OQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.741255592 (493h12m31.612338627s) [noise=true , address=145.79.227.35:4481 , piece_id=SDCEQ6ZUPWOQWZAXDARIFSCE2E523JAACJI3V4B2HIVK4MVTJ5GA, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.746731880 (493h12m31.617814925s) [noise=true , address=145.79.227.36:4519 , piece_id=CQUS7OVDXTCNBJOBGRANP6OI5Z575V5LIYWL7ITC33BASV3NVBNQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.749981779 (493h12m31.621064814s) [noise=true , address=145.79.227.15:4876 , piece_id=A6JLSZFAL6BWJMGYF2P4ED4BKPSJVTZRP4RTTZJ5Q773VKU7RMGA, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.753559871 (493h12m31.624642916s) [noise=true , address=145.79.227.38:4979 , piece_id=PQJOGLBHDAHRLJLX7S7EFCJKYCRNW6OUXASOGNP2TRAUUSECLXJA, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.756206361 (493h12m31.627289406s) [noise=true , address=145.79.227.11:4731 , piece_id=V2KTGIIB6AMNPXGIC42Y7SZSXUIT4RQ4OAU46KKTNPUWYPQ4QCSA, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.759913300 (493h12m31.630996335s) [noise=true , address=145.79.227.48:4302 , piece_id=S3NG7NAHEXKU46IXBMD63RQKCTRILGOZECX2XD2A3ROA2PBIGYGQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.768062577 (493h12m31.639145653s) [noise=true , address=145.79.227.14:4837 , piece_id=JDDVPYT7EQI2P76KPRZOPRJUUUTW2VRJ6KMPJGAWI7MV2BO3NKPA, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.787970264 (493h12m31.659053299s) [noise=true , address=145.79.227.52:4446 , piece_id=V76JR4LXHHQQYI4QIN64STDQ2O6S6535UONVQFY6HC4R3FHSAGDQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.986011059 (493h12m31.857094104s) [noise=true , address=45.91.116.10:4082 , piece_id=A74HNRTT2GZS2IUELQGOI3WGXZEOPHDGQFQWIFB53D7XMTFIUWSQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.988915048 (493h12m31.859998093s) [noise=true , address=45.91.116.4:4937 , piece_id=JBURMCQXIOSR5CZ3G6KTQGTENUKBCCDJY5WEYYF6KNJO3XQIS3JQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:32.098108061 (493h12m31.969191096s) [noise=true , address=45.91.116.5:4300 , piece_id=6TUUBT2BEYSINKNNEV7URLMSMSRLPXRYNLX6QDI42JS2PC6YP4AQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:32.106526140 (493h12m31.977609176s) [noise=true , address=45.91.116.12:4226 , piece_id=Y2H2AQVG4DPT7AHYV4OUXJQF3RS55AEVA72HQ4PZOWW2A4MXBOLA, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:32.106548023 (493h12m31.977631058s) [seg_pos={0 0}, upload=432886]: Segment reached optimal threshold of 13 pieces. Dec 26 04:41:32.106762751 (493h12m31.977845786s) [noise=true , address=45.91.116.6:4973 , piece_id=NHCX3W2VS6OXRJ737XJNDUF5ZIXTKPX34M7T42OMTGDDBA2BV4FQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 04:41:32.106769504 (493h12m31.977852549s) [noise=true , address=45.91.116.6:4973 , piece_id=NHCX3W2VS6OXRJ737XJNDUF5ZIXTKPX34M7T42OMTGDDBA2BV4FQ, seg_pos={0 0}, upload=432886]: Long tail cancellation due to optimal threshold Dec 26 04:41:32.107044458 (493h12m31.978127493s) [noise=true , address=45.91.116.14:4189 , piece_id=G7UMR5C5XDWPQ3IAURKZBUEFZ3QXIOL4BDHKZJCEM56OAJAG6PQQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:60826->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:60826->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60826->45.91.116.14:4189: use of closed network connection Dec 26 04:41:32.107085475 (493h12m31.978168511s) [noise=true , address=45.91.116.14:4189 , piece_id=G7UMR5C5XDWPQ3IAURKZBUEFZ3QXIOL4BDHKZJCEM56OAJAG6PQQ, seg_pos={0 0}, upload=432886]: Long tail cancellation due to optimal threshold Dec 26 04:41:32.107097068 (493h12m31.978180143s) [noise=true , address=45.91.116.8:4047 , piece_id=FM4QI4AOMK4OYJHYHZFCJ6DNZ7X5LOZWVKT4M3QVQMOYA2OGBPVQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 04:41:32.107111876 (493h12m31.978194941s) [noise=true , address=45.91.116.8:4047 , piece_id=FM4QI4AOMK4OYJHYHZFCJ6DNZ7X5LOZWVKT4M3QVQMOYA2OGBPVQ, seg_pos={0 0}, upload=432886]: Long tail cancellation due to optimal threshold Dec 26 04:41:32.107156521 (493h12m31.978239556s) [seg_pos={0 0}, upload=432886]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:41:32.107299813 (493h12m31.978382859s) [upload=432886]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:41:32.107308611 (493h12m31.978391656s) [upload=432886]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:41:32.387333997 (493h12m32.258417043s) [upload=432886]: Done uploading segments... Dec 26 04:41:32.387359707 (493h12m32.258442742s) [upload=432886]: Upload finished. err: <nil> Duration: 6.295737236s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 29.350468378s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:42:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 29.350497955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..................................X..X..X.X....XX.X.X.X.XX.X] duration: 30.003713051s trace: https://jaeger.dev.storj.io/trace/6752deb4cacb681b FAIL (2025-12-26 04:41:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: A74HNRTT2GZS2IUELQGOI3WGXZEOPHDGQFQWIFB53D7XMTFIUWSQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 6TUUBT2BEYSINKNNEV7URLMSMSRLPXRYNLX6QDI42JS2PC6YP4AQ) context deadline exceeded ======LOGS====== 04:41:29.493415615 (493h12m29.36449865s) [upload=432886]: Starting upload Dec 26 04:41:29.493480528 (493h12m29.364563563s) [upload=432886]: Done starting upload Dec 26 04:41:29.493610946 (493h12m29.364693982s) [upload=432886]: Uploading segments... Dec 26 04:41:29.493715966 (493h12m29.364799002s) [upload=432886]: Got next segment. Inline: false Dec 26 04:41:29.493759900 (493h12m29.364842975s) [upload=432886]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:41:29.493766834 (493h12m29.364849879s) [upload=432886]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:41:29.734693269 (493h12m29.605776314s) [seg_pos={0 0}, upload=432886]: Begin upload segment... Dec 26 04:41:29.734747452 (493h12m29.605830497s) [seg_pos={0 0}, upload=432886]: Done begin upload segment. Dec 26 04:41:29.734803579 (493h12m29.605886614s) [noise=true , address=45.91.116.12:4226 , piece_id=Y2H2AQVG4DPT7AHYV4OUXJQF3RS55AEVA72HQ4PZOWW2A4MXBOLA, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.734816253 (493h12m29.605899288s) [noise=true , address=145.79.227.14:4837 , piece_id=JDDVPYT7EQI2P76KPRZOPRJUUUTW2VRJ6KMPJGAWI7MV2BO3NKPA, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.735188372 (493h12m29.606271407s) [noise=true , address=45.91.116.14:4189 , piece_id=G7UMR5C5XDWPQ3IAURKZBUEFZ3QXIOL4BDHKZJCEM56OAJAG6PQQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.735456543 (493h12m29.606539578s) [noise=true , address=145.79.227.35:4481 , piece_id=SDCEQ6ZUPWOQWZAXDARIFSCE2E523JAACJI3V4B2HIVK4MVTJ5GA, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.735759158 (493h12m29.606842193s) [noise=true , address=45.91.116.5:4300 , piece_id=6TUUBT2BEYSINKNNEV7URLMSMSRLPXRYNLX6QDI42JS2PC6YP4AQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.736049692 (493h12m29.607132737s) [noise=true , address=45.91.116.8:4047 , piece_id=FM4QI4AOMK4OYJHYHZFCJ6DNZ7X5LOZWVKT4M3QVQMOYA2OGBPVQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.737215782 (493h12m29.608298847s) [noise=true , address=45.91.116.10:4082 , piece_id=A74HNRTT2GZS2IUELQGOI3WGXZEOPHDGQFQWIFB53D7XMTFIUWSQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.737697079 (493h12m29.608780124s) [noise=true , address=145.79.227.52:4446 , piece_id=V76JR4LXHHQQYI4QIN64STDQ2O6S6535UONVQFY6HC4R3FHSAGDQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.738902164 (493h12m29.609985199s) [noise=true , address=145.79.227.45:4231 , piece_id=S6OSKDK3JHMMPY4SIBDYUOIJ3RYRDEJWLA2EHK5HYNVOYHIN35OQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.740154719 (493h12m29.611237754s) [noise=true , address=45.91.116.4:4937 , piece_id=JBURMCQXIOSR5CZ3G6KTQGTENUKBCCDJY5WEYYF6KNJO3XQIS3JQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.741360554 (493h12m29.6124436s) [noise=true , address=145.79.227.15:4876 , piece_id=A6JLSZFAL6BWJMGYF2P4ED4BKPSJVTZRP4RTTZJ5Q773VKU7RMGA, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.743538121 (493h12m29.614621166s) [noise=true , address=145.79.227.38:4979 , piece_id=PQJOGLBHDAHRLJLX7S7EFCJKYCRNW6OUXASOGNP2TRAUUSECLXJA, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.745592874 (493h12m29.616675919s) [noise=true , address=145.79.227.11:4731 , piece_id=V2KTGIIB6AMNPXGIC42Y7SZSXUIT4RQ4OAU46KKTNPUWYPQ4QCSA, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.745716649 (493h12m29.616799684s) [noise=true , address=145.79.227.36:4519 , piece_id=CQUS7OVDXTCNBJOBGRANP6OI5Z575V5LIYWL7ITC33BASV3NVBNQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.747222337 (493h12m29.618305372s) [noise=true , address=145.79.227.48:4302 , piece_id=S3NG7NAHEXKU46IXBMD63RQKCTRILGOZECX2XD2A3ROA2PBIGYGQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:29.747889948 (493h12m29.618973003s) [noise=true , address=45.91.116.6:4973 , piece_id=NHCX3W2VS6OXRJ737XJNDUF5ZIXTKPX34M7T42OMTGDDBA2BV4FQ, seg_pos={0 0}, upload=432886]: Uploading piece... Dec 26 04:41:31.347530431 (493h12m31.218613466s) [upload=432886]: Next returned nil segment Dec 26 04:41:31.347550729 (493h12m31.218633764s) [upload=432886]: Waiting for error group managing segments... Dec 26 04:41:31.732161215 (493h12m31.60324425s) [noise=true , address=145.79.227.45:4231 , piece_id=S6OSKDK3JHMMPY4SIBDYUOIJ3RYRDEJWLA2EHK5HYNVOYHIN35OQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.741255592 (493h12m31.612338627s) [noise=true , address=145.79.227.35:4481 , piece_id=SDCEQ6ZUPWOQWZAXDARIFSCE2E523JAACJI3V4B2HIVK4MVTJ5GA, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.746731880 (493h12m31.617814925s) [noise=true , address=145.79.227.36:4519 , piece_id=CQUS7OVDXTCNBJOBGRANP6OI5Z575V5LIYWL7ITC33BASV3NVBNQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.749981779 (493h12m31.621064814s) [noise=true , address=145.79.227.15:4876 , piece_id=A6JLSZFAL6BWJMGYF2P4ED4BKPSJVTZRP4RTTZJ5Q773VKU7RMGA, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.753559871 (493h12m31.624642916s) [noise=true , address=145.79.227.38:4979 , piece_id=PQJOGLBHDAHRLJLX7S7EFCJKYCRNW6OUXASOGNP2TRAUUSECLXJA, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.756206361 (493h12m31.627289406s) [noise=true , address=145.79.227.11:4731 , piece_id=V2KTGIIB6AMNPXGIC42Y7SZSXUIT4RQ4OAU46KKTNPUWYPQ4QCSA, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.759913300 (493h12m31.630996335s) [noise=true , address=145.79.227.48:4302 , piece_id=S3NG7NAHEXKU46IXBMD63RQKCTRILGOZECX2XD2A3ROA2PBIGYGQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.768062577 (493h12m31.639145653s) [noise=true , address=145.79.227.14:4837 , piece_id=JDDVPYT7EQI2P76KPRZOPRJUUUTW2VRJ6KMPJGAWI7MV2BO3NKPA, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.787970264 (493h12m31.659053299s) [noise=true , address=145.79.227.52:4446 , piece_id=V76JR4LXHHQQYI4QIN64STDQ2O6S6535UONVQFY6HC4R3FHSAGDQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.986011059 (493h12m31.857094104s) [noise=true , address=45.91.116.10:4082 , piece_id=A74HNRTT2GZS2IUELQGOI3WGXZEOPHDGQFQWIFB53D7XMTFIUWSQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:31.988915048 (493h12m31.859998093s) [noise=true , address=45.91.116.4:4937 , piece_id=JBURMCQXIOSR5CZ3G6KTQGTENUKBCCDJY5WEYYF6KNJO3XQIS3JQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:32.098108061 (493h12m31.969191096s) [noise=true , address=45.91.116.5:4300 , piece_id=6TUUBT2BEYSINKNNEV7URLMSMSRLPXRYNLX6QDI42JS2PC6YP4AQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:32.106526140 (493h12m31.977609176s) [noise=true , address=45.91.116.12:4226 , piece_id=Y2H2AQVG4DPT7AHYV4OUXJQF3RS55AEVA72HQ4PZOWW2A4MXBOLA, seg_pos={0 0}, upload=432886]: Done uploading piece. err: <nil> Dec 26 04:41:32.106548023 (493h12m31.977631058s) [seg_pos={0 0}, upload=432886]: Segment reached optimal threshold of 13 pieces. Dec 26 04:41:32.106762751 (493h12m31.977845786s) [noise=true , address=45.91.116.6:4973 , piece_id=NHCX3W2VS6OXRJ737XJNDUF5ZIXTKPX34M7T42OMTGDDBA2BV4FQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 26 04:41:32.106769504 (493h12m31.977852549s) [noise=true , address=45.91.116.6:4973 , piece_id=NHCX3W2VS6OXRJ737XJNDUF5ZIXTKPX34M7T42OMTGDDBA2BV4FQ, seg_pos={0 0}, upload=432886]: Long tail cancellation due to optimal threshold Dec 26 04:41:32.107044458 (493h12m31.978127493s) [noise=true , address=45.91.116.14:4189 , piece_id=G7UMR5C5XDWPQ3IAURKZBUEFZ3QXIOL4BDHKZJCEM56OAJAG6PQQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:60826->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:60826->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60826->45.91.116.14:4189: use of closed network connection Dec 26 04:41:32.107085475 (493h12m31.978168511s) [noise=true , address=45.91.116.14:4189 , piece_id=G7UMR5C5XDWPQ3IAURKZBUEFZ3QXIOL4BDHKZJCEM56OAJAG6PQQ, seg_pos={0 0}, upload=432886]: Long tail cancellation due to optimal threshold Dec 26 04:41:32.107097068 (493h12m31.978180143s) [noise=true , address=45.91.116.8:4047 , piece_id=FM4QI4AOMK4OYJHYHZFCJ6DNZ7X5LOZWVKT4M3QVQMOYA2OGBPVQ, seg_pos={0 0}, upload=432886]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 04:41:32.107111876 (493h12m31.978194941s) [noise=true , address=45.91.116.8:4047 , piece_id=FM4QI4AOMK4OYJHYHZFCJ6DNZ7X5LOZWVKT4M3QVQMOYA2OGBPVQ, seg_pos={0 0}, upload=432886]: Long tail cancellation due to optimal threshold Dec 26 04:41:32.107156521 (493h12m31.978239556s) [seg_pos={0 0}, upload=432886]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:41:32.107299813 (493h12m31.978382859s) [upload=432886]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:41:32.107308611 (493h12m31.978391656s) [upload=432886]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:41:32.387333997 (493h12m32.258417043s) [upload=432886]: Done uploading segments... Dec 26 04:41:32.387359707 (493h12m32.258442742s) [upload=432886]: Upload finished. err: <nil> Duration: 30.003737035s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.971527157s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:41:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.971551715s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 28.082923892s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:41:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 28.082947345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...................................X..X..X.X....XX.X.X.X.XX.] duration: 5.762756719s trace: https://jaeger.dev.storj.io/trace/2bc989999d88e37d FAIL (2025-12-26 04:40:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: ERGHDY74JWTJ5SFS2PYPO3SEWSG6FALKJQPIHWBCSB7EDAPFYATQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: OMASWASBXUYIIINFJLOC4SVEJZ6KQNYY5YIOUB2V4M4IUKSPYF5Q) context deadline exceeded ======LOGS====== Dec 26 04:40:21.200815884 (493h11m21.071898919s) [upload=432868]: Starting upload Dec 26 04:40:21.200892760 (493h11m21.071975796s) [upload=432868]: Done starting upload Dec 26 04:40:21.201057575 (493h11m21.07214062s) [upload=432868]: Uploading segments... Dec 26 04:40:21.201301409 (493h11m21.072384454s) [upload=432868]: Got next segment. Inline: false Dec 26 04:40:21.201345202 (493h11m21.072428247s) [upload=432868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:40:21.201352026 (493h11m21.072435071s) [upload=432868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:40:21.442722868 (493h11m21.313805913s) [seg_pos={0 0}, upload=432868]: Begin upload segment... Dec 26 04:40:21.442783493 (493h11m21.313866539s) [seg_pos={0 0}, upload=432868]: Done begin upload segment. Dec 26 04:40:21.442811927 (493h11m21.313894972s) [noise=true , address=45.91.116.4:4940 , piece_id=2WANR2NXYCPIRSS6GPSYNLB54DMVEMACSB5X4RALM37LHU23HBWA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.442824271 (493h11m21.313907316s) [noise=true , address=145.79.227.47:4265 , piece_id=QBUASCE4CTVO5IF622AULAMSHDYH4EJTXODHGJ5NJWDLTLLE6TLA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.443311689 (493h11m21.314394724s) [noise=true , address=145.79.227.49:4339 , piece_id=GFIYK23BVW5HNEBICHDGCDSBLTGKBS6NTGIDEPU7MIF2DGPRDUFQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.443628182 (493h11m21.314711217s) [noise=true , address=145.79.227.6:4259 , piece_id=2IABI6WJZJZHJYP2JEEENPTM4WTNSBSLCTG5IQAUUBV4H3VKGFOA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.443976685 (493h11m21.31505972s) [noise=true , address=145.79.227.16:4911 , piece_id=FDGEUHSIYYIXMKCL633ROGXMR54G6V7QQP4W6YC4GE7EHARMARHA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.444328465 (493h11m21.3154115s) [noise=true , address=45.91.116.10:4081 , piece_id=OMASWASBXUYIIINFJLOC4SVEJZ6KQNYY5YIOUB2V4M4IUKSPYF5Q, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.444335929 (493h11m21.315418974s) [noise=true , address=145.79.227.11:4731 , piece_id=7L3CAX2M2P46KBQFBIVPJATBHWAJF6TQR7TT5TPSSQJIQWGSNNCQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.444709651 (493h11m21.315792696s) [noise=true , address=145.79.227.8:4329 , piece_id=SQMYHVZI4ZLPEPHHR7RKP4WNYFCYO2FZE5POSEFNNLN27ZRYBZOQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.444782349 (493h11m21.315865394s) [noise=true , address=145.79.227.46:4602 , piece_id=QDLB5TU5EEIEOKCR7NHU6H6JZDIO3QMQJLYBW22KISVLFFRGD3QQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.445100715 (493h11m21.31618376s) [noise=true , address=45.91.116.5:4298 , piece_id=33SBITOQHBU45R74EJE7GDXAWH5AJPAIXF6476ALY2AXSGCGJ3AA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.445149127 (493h11m21.316232172s) [noise=true , address=145.79.227.15:4873 , piece_id=2AEGYMC735MU3NXDJJRWNGAXLHDQ74XIGEP2PV6DWT2YXDPZNCVQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.445459338 (493h11m21.316542403s) [noise=true , address=77.90.31.18:4805 , piece_id=AGTKXD6D7JGDD44ZBZ3DTWMZUI4WHTXSACIEFAFHEZCUKDFP4HRQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.445469438 (493h11m21.316552483s) [noise=true , address=145.79.227.39:4013 , piece_id=U5TGNNQOPZYNLEE5ZR5DLPN55YZ2U7QYEDTLRVCC7WVQB5H5G6BQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.445787994 (493h11m21.316871029s) [noise=true , address=145.79.227.42:4121 , piece_id=KE7WBC2BGE7BPZ3V7DNCPGIPBFHRVICRO25J7IEF3OJM2Q52H7WA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.446163710 (493h11m21.317246755s) [noise=true , address=145.79.227.17:4972 , piece_id=AJNK4JINXLZLD6MXD3CCEIVOBJM7U76RTGFMDRIJH2DKEY6FQQXA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.446213855 (493h11m21.31729689s) [noise=true , address=45.91.116.14:4192 , piece_id=ERGHDY74JWTJ5SFS2PYPO3SEWSG6FALKJQPIHWBCSB7EDAPFYATQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:22.619258487 (493h11m22.490341542s) [noise=true , address=77.90.31.18:4805 , piece_id=AGTKXD6D7JGDD44ZBZ3DTWMZUI4WHTXSACIEFAFHEZCUKDFP4HRQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 04:40:23.051381243 (493h11m22.922464289s) [upload=432868]: Next returned nil segment Dec 26 04:40:23.051398176 (493h11m22.922481211s) [upload=432868]: Waiting for error group managing segments... Dec 26 04:40:23.436128139 (493h11m23.307211185s) [noise=true , address=145.79.227.49:4339 , piece_id=GFIYK23BVW5HNEBICHDGCDSBLTGKBS6NTGIDEPU7MIF2DGPRDUFQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.442149315 (493h11m23.31323235s) [noise=true , address=145.79.227.17:4972 , piece_id=AJNK4JINXLZLD6MXD3CCEIVOBJM7U76RTGFMDRIJH2DKEY6FQQXA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.443200506 (493h11m23.314283551s) [noise=true , address=145.79.227.46:4602 , piece_id=QDLB5TU5EEIEOKCR7NHU6H6JZDIO3QMQJLYBW22KISVLFFRGD3QQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.446891845 (493h11m23.31797488s) [noise=true , address=145.79.227.8:4329 , piece_id=SQMYHVZI4ZLPEPHHR7RKP4WNYFCYO2FZE5POSEFNNLN27ZRYBZOQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.451341288 (493h11m23.322424333s) [noise=true , address=145.79.227.16:4911 , piece_id=FDGEUHSIYYIXMKCL633ROGXMR54G6V7QQP4W6YC4GE7EHARMARHA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.452085835 (493h11m23.32316888s) [noise=true , address=145.79.227.15:4873 , piece_id=2AEGYMC735MU3NXDJJRWNGAXLHDQ74XIGEP2PV6DWT2YXDPZNCVQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.452900727 (493h11m23.323983792s) [noise=true , address=145.79.227.6:4259 , piece_id=2IABI6WJZJZHJYP2JEEENPTM4WTNSBSLCTG5IQAUUBV4H3VKGFOA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.455542808 (493h11m23.326625853s) [noise=true , address=145.79.227.11:4731 , piece_id=7L3CAX2M2P46KBQFBIVPJATBHWAJF6TQR7TT5TPSSQJIQWGSNNCQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.461626843 (493h11m23.332709878s) [noise=true , address=145.79.227.39:4013 , piece_id=U5TGNNQOPZYNLEE5ZR5DLPN55YZ2U7QYEDTLRVCC7WVQB5H5G6BQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.464712799 (493h11m23.335795845s) [noise=true , address=145.79.227.47:4265 , piece_id=QBUASCE4CTVO5IF622AULAMSHDYH4EJTXODHGJ5NJWDLTLLE6TLA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.477931881 (493h11m23.349014926s) [noise=true , address=145.79.227.42:4121 , piece_id=KE7WBC2BGE7BPZ3V7DNCPGIPBFHRVICRO25J7IEF3OJM2Q52H7WA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.700639555 (493h11m23.5717226s) [noise=true , address=45.91.116.14:4192 , piece_id=ERGHDY74JWTJ5SFS2PYPO3SEWSG6FALKJQPIHWBCSB7EDAPFYATQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.737491437 (493h11m23.608574492s) [noise=true , address=45.91.116.10:4081 , piece_id=OMASWASBXUYIIINFJLOC4SVEJZ6KQNYY5YIOUB2V4M4IUKSPYF5Q, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.737517838 (493h11m23.608600883s) [seg_pos={0 0}, upload=432868]: Segment reached optimal threshold of 13 pieces. Dec 26 04:40:23.737774506 (493h11m23.608857591s) [noise=true , address=45.91.116.4:4940 , piece_id=2WANR2NXYCPIRSS6GPSYNLB54DMVEMACSB5X4RALM37LHU23HBWA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 04:40:23.737789475 (493h11m23.60887255s) [noise=true , address=45.91.116.4:4940 , piece_id=2WANR2NXYCPIRSS6GPSYNLB54DMVEMACSB5X4RALM37LHU23HBWA, seg_pos={0 0}, upload=432868]: Long tail cancellation due to optimal threshold Dec 26 04:40:23.738066322 (493h11m23.609149357s) [noise=true , address=45.91.116.5:4298 , piece_id=33SBITOQHBU45R74EJE7GDXAWH5AJPAIXF6476ALY2AXSGCGJ3AA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39298->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39298->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39298->45.91.116.5:4298: use of closed network connection Dec 26 04:40:23.738082713 (493h11m23.609165758s) [noise=true , address=45.91.116.5:4298 , piece_id=33SBITOQHBU45R74EJE7GDXAWH5AJPAIXF6476ALY2AXSGCGJ3AA, seg_pos={0 0}, upload=432868]: Long tail cancellation due to optimal threshold Dec 26 04:40:23.738107300 (493h11m23.609190335s) [seg_pos={0 0}, upload=432868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:40:23.738239151 (493h11m23.609322196s) [upload=432868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:40:23.738246966 (493h11m23.609330001s) [upload=432868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:40:24.031319127 (493h11m23.902402182s) [upload=432868]: Done uploading segments... Dec 26 04:40:24.031344474 (493h11m23.90242752s) [upload=432868]: Upload finished. err: <nil> Duration: 5.762775294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....................................X..X..X.X....XX.X.X.X.XX] duration: 30.004504757s trace: https://jaeger.dev.storj.io/trace/2bc989999d88e37d FAIL (2025-12-26 04:40:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: ERGHDY74JWTJ5SFS2PYPO3SEWSG6FALKJQPIHWBCSB7EDAPFYATQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: OMASWASBXUYIIINFJLOC4SVEJZ6KQNYY5YIOUB2V4M4IUKSPYF5Q) context deadline exceeded ======LOGS====== Dec 26 04:40:21.200815884 (493h11m21.071898919s) [upload=432868]: Starting upload Dec 26 04:40:21.200892760 (493h11m21.071975796s) [upload=432868]: Done starting upload Dec 26 04:40:21.201057575 (493h11m21.07214062s) [upload=432868]: Uploading segments... Dec 26 04:40:21.201301409 (493h11m21.072384454s) [upload=432868]: Got next segment. Inline: false Dec 26 04:40:21.201345202 (493h11m21.072428247s) [upload=432868]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:40:21.201352026 (493h11m21.072435071s) [upload=432868]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:40:21.442722868 (493h11m21.313805913s) [seg_pos={0 0}, upload=432868]: Begin upload segment... Dec 26 04:40:21.442783493 (493h11m21.313866539s) [seg_pos={0 0}, upload=432868]: Done begin upload segment. Dec 26 04:40:21.442811927 (493h11m21.313894972s) [noise=true , address=45.91.116.4:4940 , piece_id=2WANR2NXYCPIRSS6GPSYNLB54DMVEMACSB5X4RALM37LHU23HBWA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.442824271 (493h11m21.313907316s) [noise=true , address=145.79.227.47:4265 , piece_id=QBUASCE4CTVO5IF622AULAMSHDYH4EJTXODHGJ5NJWDLTLLE6TLA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.443311689 (493h11m21.314394724s) [noise=true , address=145.79.227.49:4339 , piece_id=GFIYK23BVW5HNEBICHDGCDSBLTGKBS6NTGIDEPU7MIF2DGPRDUFQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.443628182 (493h11m21.314711217s) [noise=true , address=145.79.227.6:4259 , piece_id=2IABI6WJZJZHJYP2JEEENPTM4WTNSBSLCTG5IQAUUBV4H3VKGFOA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.443976685 (493h11m21.31505972s) [noise=true , address=145.79.227.16:4911 , piece_id=FDGEUHSIYYIXMKCL633ROGXMR54G6V7QQP4W6YC4GE7EHARMARHA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.444328465 (493h11m21.3154115s) [noise=true , address=45.91.116.10:4081 , piece_id=OMASWASBXUYIIINFJLOC4SVEJZ6KQNYY5YIOUB2V4M4IUKSPYF5Q, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.444335929 (493h11m21.315418974s) [noise=true , address=145.79.227.11:4731 , piece_id=7L3CAX2M2P46KBQFBIVPJATBHWAJF6TQR7TT5TPSSQJIQWGSNNCQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.444709651 (493h11m21.315792696s) [noise=true , address=145.79.227.8:4329 , piece_id=SQMYHVZI4ZLPEPHHR7RKP4WNYFCYO2FZE5POSEFNNLN27ZRYBZOQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.444782349 (493h11m21.315865394s) [noise=true , address=145.79.227.46:4602 , piece_id=QDLB5TU5EEIEOKCR7NHU6H6JZDIO3QMQJLYBW22KISVLFFRGD3QQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.445100715 (493h11m21.31618376s) [noise=true , address=45.91.116.5:4298 , piece_id=33SBITOQHBU45R74EJE7GDXAWH5AJPAIXF6476ALY2AXSGCGJ3AA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.445149127 (493h11m21.316232172s) [noise=true , address=145.79.227.15:4873 , piece_id=2AEGYMC735MU3NXDJJRWNGAXLHDQ74XIGEP2PV6DWT2YXDPZNCVQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.445459338 (493h11m21.316542403s) [noise=true , address=77.90.31.18:4805 , piece_id=AGTKXD6D7JGDD44ZBZ3DTWMZUI4WHTXSACIEFAFHEZCUKDFP4HRQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.445469438 (493h11m21.316552483s) [noise=true , address=145.79.227.39:4013 , piece_id=U5TGNNQOPZYNLEE5ZR5DLPN55YZ2U7QYEDTLRVCC7WVQB5H5G6BQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.445787994 (493h11m21.316871029s) [noise=true , address=145.79.227.42:4121 , piece_id=KE7WBC2BGE7BPZ3V7DNCPGIPBFHRVICRO25J7IEF3OJM2Q52H7WA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.446163710 (493h11m21.317246755s) [noise=true , address=145.79.227.17:4972 , piece_id=AJNK4JINXLZLD6MXD3CCEIVOBJM7U76RTGFMDRIJH2DKEY6FQQXA, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:21.446213855 (493h11m21.31729689s) [noise=true , address=45.91.116.14:4192 , piece_id=ERGHDY74JWTJ5SFS2PYPO3SEWSG6FALKJQPIHWBCSB7EDAPFYATQ, seg_pos={0 0}, upload=432868]: Uploading piece... Dec 26 04:40:22.619258487 (493h11m22.490341542s) [noise=true , address=77.90.31.18:4805 , piece_id=AGTKXD6D7JGDD44ZBZ3DTWMZUI4WHTXSACIEFAFHEZCUKDFP4HRQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 04:40:23.051381243 (493h11m22.922464289s) [upload=432868]: Next returned nil segment Dec 26 04:40:23.051398176 (493h11m22.922481211s) [upload=432868]: Waiting for error group managing segments... Dec 26 04:40:23.436128139 (493h11m23.307211185s) [noise=true , address=145.79.227.49:4339 , piece_id=GFIYK23BVW5HNEBICHDGCDSBLTGKBS6NTGIDEPU7MIF2DGPRDUFQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.442149315 (493h11m23.31323235s) [noise=true , address=145.79.227.17:4972 , piece_id=AJNK4JINXLZLD6MXD3CCEIVOBJM7U76RTGFMDRIJH2DKEY6FQQXA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.443200506 (493h11m23.314283551s) [noise=true , address=145.79.227.46:4602 , piece_id=QDLB5TU5EEIEOKCR7NHU6H6JZDIO3QMQJLYBW22KISVLFFRGD3QQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.446891845 (493h11m23.31797488s) [noise=true , address=145.79.227.8:4329 , piece_id=SQMYHVZI4ZLPEPHHR7RKP4WNYFCYO2FZE5POSEFNNLN27ZRYBZOQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.451341288 (493h11m23.322424333s) [noise=true , address=145.79.227.16:4911 , piece_id=FDGEUHSIYYIXMKCL633ROGXMR54G6V7QQP4W6YC4GE7EHARMARHA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.452085835 (493h11m23.32316888s) [noise=true , address=145.79.227.15:4873 , piece_id=2AEGYMC735MU3NXDJJRWNGAXLHDQ74XIGEP2PV6DWT2YXDPZNCVQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.452900727 (493h11m23.323983792s) [noise=true , address=145.79.227.6:4259 , piece_id=2IABI6WJZJZHJYP2JEEENPTM4WTNSBSLCTG5IQAUUBV4H3VKGFOA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.455542808 (493h11m23.326625853s) [noise=true , address=145.79.227.11:4731 , piece_id=7L3CAX2M2P46KBQFBIVPJATBHWAJF6TQR7TT5TPSSQJIQWGSNNCQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.461626843 (493h11m23.332709878s) [noise=true , address=145.79.227.39:4013 , piece_id=U5TGNNQOPZYNLEE5ZR5DLPN55YZ2U7QYEDTLRVCC7WVQB5H5G6BQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.464712799 (493h11m23.335795845s) [noise=true , address=145.79.227.47:4265 , piece_id=QBUASCE4CTVO5IF622AULAMSHDYH4EJTXODHGJ5NJWDLTLLE6TLA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.477931881 (493h11m23.349014926s) [noise=true , address=145.79.227.42:4121 , piece_id=KE7WBC2BGE7BPZ3V7DNCPGIPBFHRVICRO25J7IEF3OJM2Q52H7WA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.700639555 (493h11m23.5717226s) [noise=true , address=45.91.116.14:4192 , piece_id=ERGHDY74JWTJ5SFS2PYPO3SEWSG6FALKJQPIHWBCSB7EDAPFYATQ, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.737491437 (493h11m23.608574492s) [noise=true , address=45.91.116.10:4081 , piece_id=OMASWASBXUYIIINFJLOC4SVEJZ6KQNYY5YIOUB2V4M4IUKSPYF5Q, seg_pos={0 0}, upload=432868]: Done uploading piece. err: <nil> Dec 26 04:40:23.737517838 (493h11m23.608600883s) [seg_pos={0 0}, upload=432868]: Segment reached optimal threshold of 13 pieces. Dec 26 04:40:23.737774506 (493h11m23.608857591s) [noise=true , address=45.91.116.4:4940 , piece_id=2WANR2NXYCPIRSS6GPSYNLB54DMVEMACSB5X4RALM37LHU23HBWA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 04:40:23.737789475 (493h11m23.60887255s) [noise=true , address=45.91.116.4:4940 , piece_id=2WANR2NXYCPIRSS6GPSYNLB54DMVEMACSB5X4RALM37LHU23HBWA, seg_pos={0 0}, upload=432868]: Long tail cancellation due to optimal threshold Dec 26 04:40:23.738066322 (493h11m23.609149357s) [noise=true , address=45.91.116.5:4298 , piece_id=33SBITOQHBU45R74EJE7GDXAWH5AJPAIXF6476ALY2AXSGCGJ3AA, seg_pos={0 0}, upload=432868]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39298->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39298->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39298->45.91.116.5:4298: use of closed network connection Dec 26 04:40:23.738082713 (493h11m23.609165758s) [noise=true , address=45.91.116.5:4298 , piece_id=33SBITOQHBU45R74EJE7GDXAWH5AJPAIXF6476ALY2AXSGCGJ3AA, seg_pos={0 0}, upload=432868]: Long tail cancellation due to optimal threshold Dec 26 04:40:23.738107300 (493h11m23.609190335s) [seg_pos={0 0}, upload=432868]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:40:23.738239151 (493h11m23.609322196s) [upload=432868]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:40:23.738246966 (493h11m23.609330001s) [upload=432868]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:40:24.031319127 (493h11m23.902402182s) [upload=432868]: Done uploading segments... Dec 26 04:40:24.031344474 (493h11m23.90242752s) [upload=432868]: Upload finished. err: <nil> Duration: 30.004516518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 34.77016409s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:40:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 34.770198136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....................................X..X..X.X....XX.X.X.X.X] duration: 30.002029383s trace: https://jaeger.dev.storj.io/trace/1394995f34bd17bb FAIL (2025-12-26 04:40:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: N3Q4W7DXHZXENWFO5ZJ7YZSJH7XH7JT3IOGLL6E5DX6PXZVJRHAA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: V6RNAMIBOVAGRUEI25IAGW5IFSWBKM4WOYXTYOG6J3FBB6YTIAZQ) context deadline exceeded ======LOGS====== Dec 26 04:39:50.950010350 (493h10m50.821093395s) [upload=432863]: Starting upload Dec 26 04:39:50.950062789 (493h10m50.821145835s) [upload=432863]: Done starting upload Dec 26 04:39:50.950188179 (493h10m50.821271214s) [upload=432863]: Uploading segments... Dec 26 04:39:50.950295112 (493h10m50.821378167s) [upload=432863]: Got next segment. Inline: false Dec 26 04:39:50.950386707 (493h10m50.821469742s) [upload=432863]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:39:50.950392067 (493h10m50.821475102s) [upload=432863]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:39:51.197994535 (493h10m51.06907757s) [seg_pos={0 0}, upload=432863]: Begin upload segment... Dec 26 04:39:51.198058327 (493h10m51.069141372s) [seg_pos={0 0}, upload=432863]: Done begin upload segment. Dec 26 04:39:51.198088384 (493h10m51.069171419s) [noise=true , address=145.79.227.35:4481 , piece_id=N62JA5A7WLVBXRCT4MUQC7XZH3OIRTVPIAK37JFT7PGD3HLU6VMQ, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.198096048 (493h10m51.069179094s) [noise=true , address=145.79.227.10:4402 , piece_id=CLNSAWJKNIGMCDC45L5SPRKWGUCBE6A3K5OREJRM73HTH4A6RIRA, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.198625927 (493h10m51.069708962s) [noise=true , address=45.91.116.9:4262 , piece_id=YMQKJPACQ7KJT5KS76LIJEV3GBYABW6AIGBRTOHLH6DR7KNVIVVQ, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.199314659 (493h10m51.070397704s) [noise=true , address=145.79.227.47:4266 , piece_id=ZDYQPYIPSR72D66SOER3N3MQ4MJRK2FZOXG77ZLPLJR7MLM44DTQ, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.199608578 (493h10m51.070691623s) [noise=true , address=77.90.31.14:4661 , piece_id=N3Q4W7DXHZXENWFO5ZJ7YZSJH7XH7JT3IOGLL6E5DX6PXZVJRHAA, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.199929099 (493h10m51.071012134s) [noise=true , address=77.90.31.17:4773 , piece_id=O6NAXMIBI3MIMARRS4XIVG5MBKX73T2WSW6FXWPRRJ27YV2FCD3A, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.200406227 (493h10m51.071489282s) [noise=true , address=77.90.31.10:4517 , piece_id=IDXPJ52BBFAHQ46N65RGFXXQ4M6E2SFPYZAJJIMBNLXGAD54VONQ, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.200494175 (493h10m51.07157721s) [noise=true , address=145.79.227.41:4085 , piece_id=BJIJ7OOAIAEEAS75JWMA26OKOICNZK7IWB7MEJOL7BJXBQMGRLJA, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.200902653 (493h10m51.071985698s) [noise=true , address=45.91.116.4:4937 , piece_id=JB3HO2BMSMQ3BZHJLHKUBKG5DOWPXAURO5N6WTHAK7HBXXQNHOFA, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.201275883 (493h10m51.072358928s) [noise=true , address=145.79.227.12:4767 , piece_id=QWJU7CEWAF5BMPHUOPSWECIDB6VPWUP3T2XSGLBCJDLWBBSLV6UA, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.201473650 (493h10m51.072556695s) [noise=true , address=145.79.227.36:4519 , piece_id=46Q23BZNRBSCJ3YCJ3TZYOX2ZSJUL6YYBG5CN37NP2XUJ7S3UPEA, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.201958133 (493h10m51.073041178s) [noise=true , address=45.91.116.5:4297 , piece_id=V6RNAMIBOVAGRUEI25IAGW5IFSWBKM4WOYXTYOG6J3FBB6YTIAZQ, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.202677893 (493h10m51.073760939s) [noise=true , address=145.79.227.11:4731 , piece_id=JSTHI6M4UNRCI5RRCP6CD5NCWDPTNAUYBKKWX44SJZ2NEWN2NM5Q, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.203206700 (493h10m51.074289745s) [noise=true , address=145.79.227.48:4301 , piece_id=YXQV2IRQAXIHKUKQ77JKDMUTGIS6O5JP2BEYGIXX3NJYSQDILRZQ, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.203633713 (493h10m51.074716748s) [noise=true , address=145.79.227.37:4565 , piece_id=6RK5VTYG2WRJAHBLGMDIYL52V5QCQ4H4E7HY656SDI35I5UY52XA, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:51.204018476 (493h10m51.075101511s) [noise=true , address=77.90.31.11:4553 , piece_id=RS3AXMQ2AZWOJ7UTEO32YVBRM7QFBT3LFSUHTZTHEIYY4W3HTVBA, seg_pos={0 0}, upload=432863]: Uploading piece... Dec 26 04:39:52.311283030 (493h10m52.182366065s) [noise=true , address=77.90.31.11:4553 , piece_id=RS3AXMQ2AZWOJ7UTEO32YVBRM7QFBT3LFSUHTZTHEIYY4W3HTVBA, seg_pos={0 0}, upload=432863]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000 Dec 26 04:39:52.810577224 (493h10m52.681660259s) [upload=432863]: Next returned nil segment Dec 26 04:39:52.810591190 (493h10m52.681674225s) [upload=432863]: Waiting for error group managing segments... Dec 26 04:39:53.196761054 (493h10m53.0678441s) [noise=true , address=145.79.227.48:4301 , piece_id=YXQV2IRQAXIHKUKQ77JKDMUTGIS6O5JP2BEYGIXX3NJYSQDILRZQ, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.197051508 (493h10m53.068134543s) [noise=true , address=145.79.227.11:4731 , piece_id=JSTHI6M4UNRCI5RRCP6CD5NCWDPTNAUYBKKWX44SJZ2NEWN2NM5Q, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.199778861 (493h10m53.070861896s) [noise=true , address=145.79.227.12:4767 , piece_id=QWJU7CEWAF5BMPHUOPSWECIDB6VPWUP3T2XSGLBCJDLWBBSLV6UA, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.205117718 (493h10m53.076200763s) [noise=true , address=145.79.227.37:4565 , piece_id=6RK5VTYG2WRJAHBLGMDIYL52V5QCQ4H4E7HY656SDI35I5UY52XA, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.205302339 (493h10m53.076385374s) [noise=true , address=145.79.227.10:4402 , piece_id=CLNSAWJKNIGMCDC45L5SPRKWGUCBE6A3K5OREJRM73HTH4A6RIRA, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.207187789 (493h10m53.078270824s) [noise=true , address=145.79.227.41:4085 , piece_id=BJIJ7OOAIAEEAS75JWMA26OKOICNZK7IWB7MEJOL7BJXBQMGRLJA, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.219729711 (493h10m53.090812756s) [noise=true , address=145.79.227.47:4266 , piece_id=ZDYQPYIPSR72D66SOER3N3MQ4MJRK2FZOXG77ZLPLJR7MLM44DTQ, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.230545416 (493h10m53.101628451s) [noise=true , address=145.79.227.35:4481 , piece_id=N62JA5A7WLVBXRCT4MUQC7XZH3OIRTVPIAK37JFT7PGD3HLU6VMQ, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.232766906 (493h10m53.103849952s) [noise=true , address=145.79.227.36:4519 , piece_id=46Q23BZNRBSCJ3YCJ3TZYOX2ZSJUL6YYBG5CN37NP2XUJ7S3UPEA, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.397996692 (493h10m53.269079737s) [noise=true , address=45.91.116.4:4937 , piece_id=JB3HO2BMSMQ3BZHJLHKUBKG5DOWPXAURO5N6WTHAK7HBXXQNHOFA, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.458229977 (493h10m53.329313012s) [noise=true , address=45.91.116.5:4297 , piece_id=V6RNAMIBOVAGRUEI25IAGW5IFSWBKM4WOYXTYOG6J3FBB6YTIAZQ, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.458642422 (493h10m53.329725467s) [noise=true , address=77.90.31.14:4661 , piece_id=N3Q4W7DXHZXENWFO5ZJ7YZSJH7XH7JT3IOGLL6E5DX6PXZVJRHAA, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.536688847 (493h10m53.407771892s) [noise=true , address=77.90.31.17:4773 , piece_id=O6NAXMIBI3MIMARRS4XIVG5MBKX73T2WSW6FXWPRRJ27YV2FCD3A, seg_pos={0 0}, upload=432863]: Done uploading piece. err: <nil> Dec 26 04:39:53.536714916 (493h10m53.407797961s) [seg_pos={0 0}, upload=432863]: Segment reached optimal threshold of 13 pieces. Dec 26 04:39:53.536947108 (493h10m53.408030183s) [noise=true , address=45.91.116.9:4262 , piece_id=YMQKJPACQ7KJT5KS76LIJEV3GBYABW6AIGBRTOHLH6DR7KNVIVVQ, seg_pos={0 0}, upload=432863]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 04:39:53.536957868 (493h10m53.408040913s) [noise=true , address=45.91.116.9:4262 , piece_id=YMQKJPACQ7KJT5KS76LIJEV3GBYABW6AIGBRTOHLH6DR7KNVIVVQ, seg_pos={0 0}, upload=432863]: Long tail cancellation due to optimal threshold Dec 26 04:39:53.537027821 (493h10m53.408110866s) [noise=true , address=77.90.31.10:4517 , piece_id=IDXPJ52BBFAHQ46N65RGFXXQ4M6E2SFPYZAJJIMBNLXGAD54VONQ, seg_pos={0 0}, upload=432863]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 26 04:39:53.537045035 (493h10m53.40812809s) [noise=true , address=77.90.31.10:4517 , piece_id=IDXPJ52BBFAHQ46N65RGFXXQ4M6E2SFPYZAJJIMBNLXGAD54VONQ, seg_pos={0 0}, upload=432863]: Long tail cancellation due to optimal threshold Dec 26 04:39:53.537070263 (493h10m53.408153308s) [seg_pos={0 0}, upload=432863]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:39:53.537170844 (493h10m53.408253889s) [upload=432863]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:39:53.537183869 (493h10m53.408266924s) [upload=432863]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:39:53.850106167 (493h10m53.721189212s) [upload=432863]: Done uploading segments... Dec 26 04:39:53.850130033 (493h10m53.721213078s) [upload=432863]: Upload finished. err: <nil> Duration: 30.002053919s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.500093476s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:40:18 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.500129846s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 28.811640083s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:40:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 28.811654792s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.204869289s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:39:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.204945924s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 23.154129554s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:39:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 23.154152077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......................................X..X..X.X....XX.X.X.X.] duration: 24.341583693s trace: https://jaeger.dev.storj.io/trace/685ae14bc0991fc9 FAIL (2025-12-26 04:39:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: EHGW6HKAKHGLFOOR742CEAESNXXRVBFR7UDMTWEW2MJGOFKZWSTA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: XTMPREQ3MCRQ4SRVNZMIZA246ZAGSBVAAWSV3IQ2PCI2JUKANK5A) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: G2BRK4KBNGPYLQG4YAC7CSL4OVXO3WJNC2FMEXKP2UULQ4J74K7Q) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: DJXJ6JSYMOF77GZQB2CZPREPD4PZXYXQLC3QZJT7SYRFDJ7TAUEA) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: 3UD2Q2S5KWMBEVR2UZYUO3LVCEYUU4GVLC7TMWW3BX23S3PKNNWA) context deadline exceeded; piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: JYRGS6Y4QMV5RXXGEDVUIQAADKGPV7WMSG4LIK6ADSQVGOX632FQ) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: DZKPA7SNITHSMKRUZV7QT7LSDH7QEFEDQCWJJNTD3LESKJMXQSLQ) context deadline exceeded ======LOGS====== Dec 26 04:37:59.406649235 (493h8m59.27773227s) [upload=432829]: Starting upload Dec 26 04:37:59.406704069 (493h8m59.277787105s) [upload=432829]: Done starting upload Dec 26 04:37:59.406834858 (493h8m59.277917893s) [upload=432829]: Uploading segments... Dec 26 04:37:59.406927935 (493h8m59.27801097s) [upload=432829]: Got next segment. Inline: false Dec 26 04:37:59.406959566 (493h8m59.278042791s) [upload=432829]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:37:59.406964425 (493h8m59.27804746s) [upload=432829]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:37:59.670630030 (493h8m59.541713065s) [seg_pos={0 0}, upload=432829]: Begin upload segment... Dec 26 04:37:59.670670668 (493h8m59.541753703s) [seg_pos={0 0}, upload=432829]: Done begin upload segment. Dec 26 04:37:59.670693120 (493h8m59.541776155s) [noise=true , address=145.79.227.49:4339 , piece_id=JYRGS6Y4QMV5RXXGEDVUIQAADKGPV7WMSG4LIK6ADSQVGOX632FQ, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.670870547 (493h8m59.541953582s) [noise=true , address=145.79.227.37:4565 , piece_id=EQ2YC7F6KRJWHMTVG7WAV3C5I7C4EV5FWVLBLAJC7XWWSCXSJZSA, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.671561082 (493h8m59.542644128s) [noise=true , address=145.79.227.16:4911 , piece_id=DZKPA7SNITHSMKRUZV7QT7LSDH7QEFEDQCWJJNTD3LESKJMXQSLQ, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.671870682 (493h8m59.542953717s) [noise=true , address=145.79.227.8:4329 , piece_id=EHGW6HKAKHGLFOOR742CEAESNXXRVBFR7UDMTWEW2MJGOFKZWSTA, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.672150024 (493h8m59.543233069s) [noise=true , address=45.91.116.10:4082 , piece_id=XTMPREQ3MCRQ4SRVNZMIZA246ZAGSBVAAWSV3IQ2PCI2JUKANK5A, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.672460385 (493h8m59.54354343s) [noise=true , address=145.79.227.36:4518 , piece_id=XMXXT2KDVLWBQWOFJGT4425IACRS333H644BQUCUQUVMIIVW45XA, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.672770906 (493h8m59.543853952s) [noise=true , address=45.91.116.13:4153 , piece_id=Q5ZBPTHZZDVM3ZXHFJSUS377RASXRBHFRA3WL67JR3BJ55IATZ2A, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.674534745 (493h8m59.54561778s) [noise=true , address=45.91.116.12:4226 , piece_id=KFSJNTYVNHKFQAGPL63DZWUQC4EVYUIDAZMPMTGLQT6PG5ROT3ZQ, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.674796293 (493h8m59.545879338s) [noise=true , address=145.79.227.48:4303 , piece_id=H2YVVRSYRG5RLD3UXT4HWGHOUJ26VECNSMC7PFMQLJSISJWAWB6Q, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.678227447 (493h8m59.549310482s) [noise=true , address=145.79.227.15:4876 , piece_id=KCHPV7S3BPCPADWFESSZTOAGISX3GA3BBWNWHARMXNYIYEH22SQQ, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.679780233 (493h8m59.550863268s) [noise=true , address=145.79.227.39:4013 , piece_id=G2BRK4KBNGPYLQG4YAC7CSL4OVXO3WJNC2FMEXKP2UULQ4J74K7Q, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.681066052 (493h8m59.552149087s) [noise=true , address=145.79.227.38:4979 , piece_id=DJXJ6JSYMOF77GZQB2CZPREPD4PZXYXQLC3QZJT7SYRFDJ7TAUEA, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.681410057 (493h8m59.552493092s) [noise=true , address=145.79.227.12:4767 , piece_id=QWBQZFK7AYK6OTUUBE37ZRJ2H52NHE2SOUJHZ7XLTAXDWOT4AT5A, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.682847134 (493h8m59.553930169s) [noise=true , address=77.90.31.16:4736 , piece_id=GDL5H2HMWD2XWAKZZUH3FJHJCK25HJJB5ZIJMY3HRFFIUDXHTQZQ, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.683442317 (493h8m59.554525362s) [noise=true , address=145.79.227.4:4187 , piece_id=2OSOYT7YDDMWIRIG3RS23NAEWJ5DC5COXKDJXU77T6C7F7RUCGKA, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.684257138 (493h8m59.555340183s) [noise=true , address=145.79.227.50:4374 , piece_id=3UD2Q2S5KWMBEVR2UZYUO3LVCEYUU4GVLC7TMWW3BX23S3PKNNWA, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:38:01.506792673 (493h9m1.377875718s) [upload=432829]: Next returned nil segment Dec 26 04:38:01.506808142 (493h9m1.377891187s) [upload=432829]: Waiting for error group managing segments... Dec 26 04:38:01.831902808 (493h9m1.702985854s) [noise=true , address=145.79.227.36:4518 , piece_id=XMXXT2KDVLWBQWOFJGT4425IACRS333H644BQUCUQUVMIIVW45XA, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.839217797 (493h9m1.710300842s) [noise=true , address=145.79.227.15:4876 , piece_id=KCHPV7S3BPCPADWFESSZTOAGISX3GA3BBWNWHARMXNYIYEH22SQQ, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.848347252 (493h9m1.719430297s) [noise=true , address=145.79.227.37:4565 , piece_id=EQ2YC7F6KRJWHMTVG7WAV3C5I7C4EV5FWVLBLAJC7XWWSCXSJZSA, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.851901630 (493h9m1.722984675s) [noise=true , address=145.79.227.49:4339 , piece_id=JYRGS6Y4QMV5RXXGEDVUIQAADKGPV7WMSG4LIK6ADSQVGOX632FQ, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.859604668 (493h9m1.730687713s) [noise=true , address=145.79.227.8:4329 , piece_id=EHGW6HKAKHGLFOOR742CEAESNXXRVBFR7UDMTWEW2MJGOFKZWSTA, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.859882176 (493h9m1.730965221s) [noise=true , address=145.79.227.50:4374 , piece_id=3UD2Q2S5KWMBEVR2UZYUO3LVCEYUU4GVLC7TMWW3BX23S3PKNNWA, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.860274934 (493h9m1.731357979s) [noise=true , address=145.79.227.38:4979 , piece_id=DJXJ6JSYMOF77GZQB2CZPREPD4PZXYXQLC3QZJT7SYRFDJ7TAUEA, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.907862159 (493h9m1.778945204s) [noise=true , address=45.91.116.13:4153 , piece_id=Q5ZBPTHZZDVM3ZXHFJSUS377RASXRBHFRA3WL67JR3BJ55IATZ2A, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.915313418 (493h9m1.786396463s) [noise=true , address=145.79.227.16:4911 , piece_id=DZKPA7SNITHSMKRUZV7QT7LSDH7QEFEDQCWJJNTD3LESKJMXQSLQ, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.931269200 (493h9m1.802352245s) [noise=true , address=45.91.116.10:4082 , piece_id=XTMPREQ3MCRQ4SRVNZMIZA246ZAGSBVAAWSV3IQ2PCI2JUKANK5A, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.946637765 (493h9m1.81772081s) [noise=true , address=145.79.227.39:4013 , piece_id=G2BRK4KBNGPYLQG4YAC7CSL4OVXO3WJNC2FMEXKP2UULQ4J74K7Q, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.995672124 (493h9m1.866755169s) [noise=true , address=77.90.31.16:4736 , piece_id=GDL5H2HMWD2XWAKZZUH3FJHJCK25HJJB5ZIJMY3HRFFIUDXHTQZQ, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:02.033417488 (493h9m1.904500543s) [noise=true , address=45.91.116.12:4226 , piece_id=KFSJNTYVNHKFQAGPL63DZWUQC4EVYUIDAZMPMTGLQT6PG5ROT3ZQ, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:02.033446122 (493h9m1.904529167s) [seg_pos={0 0}, upload=432829]: Segment reached optimal threshold of 13 pieces. Dec 26 04:38:02.033699034 (493h9m1.904782079s) [noise=true , address=145.79.227.4:4187 , piece_id=2OSOYT7YDDMWIRIG3RS23NAEWJ5DC5COXKDJXU77T6C7F7RUCGKA, seg_pos={0 0}, upload=432829]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 26 04:38:02.033709845 (493h9m1.9047929s) [noise=true , address=145.79.227.4:4187 , piece_id=2OSOYT7YDDMWIRIG3RS23NAEWJ5DC5COXKDJXU77T6C7F7RUCGKA, seg_pos={0 0}, upload=432829]: Long tail cancellation due to optimal threshold Dec 26 04:38:02.034010748 (493h9m1.905093803s) [noise=true , address=145.79.227.12:4767 , piece_id=QWBQZFK7AYK6OTUUBE37ZRJ2H52NHE2SOUJHZ7XLTAXDWOT4AT5A, seg_pos={0 0}, upload=432829]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:58762->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:58762->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58762->145.79.227.12:4767: use of closed network connection Dec 26 04:38:02.034057907 (493h9m1.905140952s) [noise=true , address=145.79.227.12:4767 , piece_id=QWBQZFK7AYK6OTUUBE37ZRJ2H52NHE2SOUJHZ7XLTAXDWOT4AT5A, seg_pos={0 0}, upload=432829]: Long tail cancellation due to optimal threshold Dec 26 04:38:02.034127810 (493h9m1.905210866s) [noise=true , address=145.79.227.48:4303 , piece_id=H2YVVRSYRG5RLD3UXT4HWGHOUJ26VECNSMC7PFMQLJSISJWAWB6Q, seg_pos={0 0}, upload=432829]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 26 04:38:02.034138010 (493h9m1.905221055s) [noise=true , address=145.79.227.48:4303 , piece_id=H2YVVRSYRG5RLD3UXT4HWGHOUJ26VECNSMC7PFMQLJSISJWAWB6Q, seg_pos={0 0}, upload=432829]: Long tail cancellation due to optimal threshold Dec 26 04:38:02.034164550 (493h9m1.905247605s) [seg_pos={0 0}, upload=432829]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:38:02.034308374 (493h9m1.905391429s) [upload=432829]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:38:02.034457538 (493h9m1.905540583s) [upload=432829]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:38:02.384349433 (493h9m2.255432478s) [upload=432829]: Done uploading segments... Dec 26 04:38:02.384372868 (493h9m2.255455903s) [upload=432829]: Upload finished. err: <nil> Duration: 24.341607118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 38.192670275s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:38:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 38.192696564s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 8.01075586s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:38:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 8.010785276s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......................................X..X..X.X....XX.X.X.X] duration: 30.003048324s trace: https://jaeger.dev.storj.io/trace/685ae14bc0991fc9 FAIL (2025-12-26 04:38:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: EHGW6HKAKHGLFOOR742CEAESNXXRVBFR7UDMTWEW2MJGOFKZWSTA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: XTMPREQ3MCRQ4SRVNZMIZA246ZAGSBVAAWSV3IQ2PCI2JUKANK5A) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: G2BRK4KBNGPYLQG4YAC7CSL4OVXO3WJNC2FMEXKP2UULQ4J74K7Q) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: DJXJ6JSYMOF77GZQB2CZPREPD4PZXYXQLC3QZJT7SYRFDJ7TAUEA) context deadline exceeded; piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: 3UD2Q2S5KWMBEVR2UZYUO3LVCEYUU4GVLC7TMWW3BX23S3PKNNWA) context deadline exceeded; piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: JYRGS6Y4QMV5RXXGEDVUIQAADKGPV7WMSG4LIK6ADSQVGOX632FQ) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: DZKPA7SNITHSMKRUZV7QT7LSDH7QEFEDQCWJJNTD3LESKJMXQSLQ) context deadline exceeded ======LOGS====== Dec 26 04:37:59.406649235 (493h8m59.27773227s) [upload=432829]: Starting upload Dec 26 04:37:59.406704069 (493h8m59.277787105s) [upload=432829]: Done starting upload Dec 26 04:37:59.406834858 (493h8m59.277917893s) [upload=432829]: Uploading segments... Dec 26 04:37:59.406927935 (493h8m59.27801097s) [upload=432829]: Got next segment. Inline: false Dec 26 04:37:59.406959566 (493h8m59.278042791s) [upload=432829]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:37:59.406964425 (493h8m59.27804746s) [upload=432829]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:37:59.670630030 (493h8m59.541713065s) [seg_pos={0 0}, upload=432829]: Begin upload segment... Dec 26 04:37:59.670670668 (493h8m59.541753703s) [seg_pos={0 0}, upload=432829]: Done begin upload segment. Dec 26 04:37:59.670693120 (493h8m59.541776155s) [noise=true , address=145.79.227.49:4339 , piece_id=JYRGS6Y4QMV5RXXGEDVUIQAADKGPV7WMSG4LIK6ADSQVGOX632FQ, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.670870547 (493h8m59.541953582s) [noise=true , address=145.79.227.37:4565 , piece_id=EQ2YC7F6KRJWHMTVG7WAV3C5I7C4EV5FWVLBLAJC7XWWSCXSJZSA, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.671561082 (493h8m59.542644128s) [noise=true , address=145.79.227.16:4911 , piece_id=DZKPA7SNITHSMKRUZV7QT7LSDH7QEFEDQCWJJNTD3LESKJMXQSLQ, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.671870682 (493h8m59.542953717s) [noise=true , address=145.79.227.8:4329 , piece_id=EHGW6HKAKHGLFOOR742CEAESNXXRVBFR7UDMTWEW2MJGOFKZWSTA, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.672150024 (493h8m59.543233069s) [noise=true , address=45.91.116.10:4082 , piece_id=XTMPREQ3MCRQ4SRVNZMIZA246ZAGSBVAAWSV3IQ2PCI2JUKANK5A, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.672460385 (493h8m59.54354343s) [noise=true , address=145.79.227.36:4518 , piece_id=XMXXT2KDVLWBQWOFJGT4425IACRS333H644BQUCUQUVMIIVW45XA, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.672770906 (493h8m59.543853952s) [noise=true , address=45.91.116.13:4153 , piece_id=Q5ZBPTHZZDVM3ZXHFJSUS377RASXRBHFRA3WL67JR3BJ55IATZ2A, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.674534745 (493h8m59.54561778s) [noise=true , address=45.91.116.12:4226 , piece_id=KFSJNTYVNHKFQAGPL63DZWUQC4EVYUIDAZMPMTGLQT6PG5ROT3ZQ, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.674796293 (493h8m59.545879338s) [noise=true , address=145.79.227.48:4303 , piece_id=H2YVVRSYRG5RLD3UXT4HWGHOUJ26VECNSMC7PFMQLJSISJWAWB6Q, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.678227447 (493h8m59.549310482s) [noise=true , address=145.79.227.15:4876 , piece_id=KCHPV7S3BPCPADWFESSZTOAGISX3GA3BBWNWHARMXNYIYEH22SQQ, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.679780233 (493h8m59.550863268s) [noise=true , address=145.79.227.39:4013 , piece_id=G2BRK4KBNGPYLQG4YAC7CSL4OVXO3WJNC2FMEXKP2UULQ4J74K7Q, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.681066052 (493h8m59.552149087s) [noise=true , address=145.79.227.38:4979 , piece_id=DJXJ6JSYMOF77GZQB2CZPREPD4PZXYXQLC3QZJT7SYRFDJ7TAUEA, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.681410057 (493h8m59.552493092s) [noise=true , address=145.79.227.12:4767 , piece_id=QWBQZFK7AYK6OTUUBE37ZRJ2H52NHE2SOUJHZ7XLTAXDWOT4AT5A, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.682847134 (493h8m59.553930169s) [noise=true , address=77.90.31.16:4736 , piece_id=GDL5H2HMWD2XWAKZZUH3FJHJCK25HJJB5ZIJMY3HRFFIUDXHTQZQ, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.683442317 (493h8m59.554525362s) [noise=true , address=145.79.227.4:4187 , piece_id=2OSOYT7YDDMWIRIG3RS23NAEWJ5DC5COXKDJXU77T6C7F7RUCGKA, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:37:59.684257138 (493h8m59.555340183s) [noise=true , address=145.79.227.50:4374 , piece_id=3UD2Q2S5KWMBEVR2UZYUO3LVCEYUU4GVLC7TMWW3BX23S3PKNNWA, seg_pos={0 0}, upload=432829]: Uploading piece... Dec 26 04:38:01.506792673 (493h9m1.377875718s) [upload=432829]: Next returned nil segment Dec 26 04:38:01.506808142 (493h9m1.377891187s) [upload=432829]: Waiting for error group managing segments... Dec 26 04:38:01.831902808 (493h9m1.702985854s) [noise=true , address=145.79.227.36:4518 , piece_id=XMXXT2KDVLWBQWOFJGT4425IACRS333H644BQUCUQUVMIIVW45XA, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.839217797 (493h9m1.710300842s) [noise=true , address=145.79.227.15:4876 , piece_id=KCHPV7S3BPCPADWFESSZTOAGISX3GA3BBWNWHARMXNYIYEH22SQQ, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.848347252 (493h9m1.719430297s) [noise=true , address=145.79.227.37:4565 , piece_id=EQ2YC7F6KRJWHMTVG7WAV3C5I7C4EV5FWVLBLAJC7XWWSCXSJZSA, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.851901630 (493h9m1.722984675s) [noise=true , address=145.79.227.49:4339 , piece_id=JYRGS6Y4QMV5RXXGEDVUIQAADKGPV7WMSG4LIK6ADSQVGOX632FQ, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.859604668 (493h9m1.730687713s) [noise=true , address=145.79.227.8:4329 , piece_id=EHGW6HKAKHGLFOOR742CEAESNXXRVBFR7UDMTWEW2MJGOFKZWSTA, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.859882176 (493h9m1.730965221s) [noise=true , address=145.79.227.50:4374 , piece_id=3UD2Q2S5KWMBEVR2UZYUO3LVCEYUU4GVLC7TMWW3BX23S3PKNNWA, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.860274934 (493h9m1.731357979s) [noise=true , address=145.79.227.38:4979 , piece_id=DJXJ6JSYMOF77GZQB2CZPREPD4PZXYXQLC3QZJT7SYRFDJ7TAUEA, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.907862159 (493h9m1.778945204s) [noise=true , address=45.91.116.13:4153 , piece_id=Q5ZBPTHZZDVM3ZXHFJSUS377RASXRBHFRA3WL67JR3BJ55IATZ2A, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.915313418 (493h9m1.786396463s) [noise=true , address=145.79.227.16:4911 , piece_id=DZKPA7SNITHSMKRUZV7QT7LSDH7QEFEDQCWJJNTD3LESKJMXQSLQ, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.931269200 (493h9m1.802352245s) [noise=true , address=45.91.116.10:4082 , piece_id=XTMPREQ3MCRQ4SRVNZMIZA246ZAGSBVAAWSV3IQ2PCI2JUKANK5A, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.946637765 (493h9m1.81772081s) [noise=true , address=145.79.227.39:4013 , piece_id=G2BRK4KBNGPYLQG4YAC7CSL4OVXO3WJNC2FMEXKP2UULQ4J74K7Q, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:01.995672124 (493h9m1.866755169s) [noise=true , address=77.90.31.16:4736 , piece_id=GDL5H2HMWD2XWAKZZUH3FJHJCK25HJJB5ZIJMY3HRFFIUDXHTQZQ, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:02.033417488 (493h9m1.904500543s) [noise=true , address=45.91.116.12:4226 , piece_id=KFSJNTYVNHKFQAGPL63DZWUQC4EVYUIDAZMPMTGLQT6PG5ROT3ZQ, seg_pos={0 0}, upload=432829]: Done uploading piece. err: <nil> Dec 26 04:38:02.033446122 (493h9m1.904529167s) [seg_pos={0 0}, upload=432829]: Segment reached optimal threshold of 13 pieces. Dec 26 04:38:02.033699034 (493h9m1.904782079s) [noise=true , address=145.79.227.4:4187 , piece_id=2OSOYT7YDDMWIRIG3RS23NAEWJ5DC5COXKDJXU77T6C7F7RUCGKA, seg_pos={0 0}, upload=432829]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 26 04:38:02.033709845 (493h9m1.9047929s) [noise=true , address=145.79.227.4:4187 , piece_id=2OSOYT7YDDMWIRIG3RS23NAEWJ5DC5COXKDJXU77T6C7F7RUCGKA, seg_pos={0 0}, upload=432829]: Long tail cancellation due to optimal threshold Dec 26 04:38:02.034010748 (493h9m1.905093803s) [noise=true , address=145.79.227.12:4767 , piece_id=QWBQZFK7AYK6OTUUBE37ZRJ2H52NHE2SOUJHZ7XLTAXDWOT4AT5A, seg_pos={0 0}, upload=432829]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:58762->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:58762->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58762->145.79.227.12:4767: use of closed network connection Dec 26 04:38:02.034057907 (493h9m1.905140952s) [noise=true , address=145.79.227.12:4767 , piece_id=QWBQZFK7AYK6OTUUBE37ZRJ2H52NHE2SOUJHZ7XLTAXDWOT4AT5A, seg_pos={0 0}, upload=432829]: Long tail cancellation due to optimal threshold Dec 26 04:38:02.034127810 (493h9m1.905210866s) [noise=true , address=145.79.227.48:4303 , piece_id=H2YVVRSYRG5RLD3UXT4HWGHOUJ26VECNSMC7PFMQLJSISJWAWB6Q, seg_pos={0 0}, upload=432829]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 26 04:38:02.034138010 (493h9m1.905221055s) [noise=true , address=145.79.227.48:4303 , piece_id=H2YVVRSYRG5RLD3UXT4HWGHOUJ26VECNSMC7PFMQLJSISJWAWB6Q, seg_pos={0 0}, upload=432829]: Long tail cancellation due to optimal threshold Dec 26 04:38:02.034164550 (493h9m1.905247605s) [seg_pos={0 0}, upload=432829]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:38:02.034308374 (493h9m1.905391429s) [upload=432829]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:38:02.034457538 (493h9m1.905540583s) [upload=432829]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:38:02.384349433 (493h9m2.255432478s) [upload=432829]: Done uploading segments... Dec 26 04:38:02.384372868 (493h9m2.255455903s) [upload=432829]: Upload finished. err: <nil> Duration: 30.003070996s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 26.737718749s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:37:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 26.737735001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................................X..X..X.X....XX.X.X.] duration: 5.700897518s trace: https://jaeger.dev.storj.io/trace/6be8f72b02990c84 FAIL (2025-12-26 04:37:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 04:36:47.735473833 (493h7m47.606556868s) [upload=432811]: Starting upload Dec 26 04:36:47.735529369 (493h7m47.606612404s) [upload=432811]: Done starting upload Dec 26 04:36:47.735604121 (493h7m47.606687156s) [upload=432811]: Uploading segments... Dec 26 04:36:47.735835983 (493h7m47.606919018s) [upload=432811]: Got next segment. Inline: false Dec 26 04:36:47.735866471 (493h7m47.606949506s) [upload=432811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:36:47.735871361 (493h7m47.606954396s) [upload=432811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:36:47.983428432 (493h7m47.854511468s) [seg_pos={0 0}, upload=432811]: Begin upload segment... Dec 26 04:36:47.983474320 (493h7m47.854557365s) [seg_pos={0 0}, upload=432811]: Done begin upload segment. Dec 26 04:36:47.983500750 (493h7m47.854583795s) [noise=true , address=145.79.227.36:4518 , piece_id=BLEKUBCF3JD6G5Z3BXBC6RZTNBUIHRZUSQZ7NYHZJN2TQXEQSVJA, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.983515428 (493h7m47.854598463s) [noise=true , address=145.79.227.47:4266 , piece_id=6BHR4XPIWMPIVMQPX6LEZSG2V4VBUBILP2743K2TLGYFHLY5X5VA, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.983598146 (493h7m47.854681181s) [noise=true , address=145.79.227.46:4602 , piece_id=ZNZ3ERQWYTJTNC3HVJ4D75BNGVWV7XBMLKROFANXU7YCJBWLIG4Q, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.984082268 (493h7m47.855165313s) [noise=true , address=145.79.227.52:4445 , piece_id=WDUBLLSJD7DZXF5RX4TKDSMCTOYJDK6YNEIYTS6PVKZZBU5LAZHQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.984402117 (493h7m47.855485152s) [noise=true , address=45.91.116.6:4974 , piece_id=LNIFEY4C5NIVJI5L2WHP7RLU2JHR7LLQEQB342U4EV2OPKLHE3HQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.984702778 (493h7m47.855785813s) [noise=true , address=45.91.116.10:4084 , piece_id=7Y3VO42IM3G6TRT4LYVFGHOCRAV5TU3Z4HIV6LYYVYMGGYRUGCDQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.985048998 (493h7m47.856132043s) [noise=true , address=145.79.227.19:4079 , piece_id=SCYS5EQ5U4XUS6KKPTOYTHRXJLDTBFRBDPRXYC7LOBYZNVWJOWYA, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.985056593 (493h7m47.856139628s) [noise=true , address=45.91.116.13:4154 , piece_id=6KNERLGIUZXTFJ6MV5GZOEVVLRCW3ODGIVVIAY5TAWSTNGJVLUZA, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.985330894 (493h7m47.856413929s) [noise=true , address=45.91.116.12:4226 , piece_id=PYNBIOFRYJ5LEUYUKQCY536NSEO2P6E5USVH6GVAE4XCSSNJRIDA, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.985425695 (493h7m47.85650873s) [noise=true , address=145.79.227.43:4158 , piece_id=7CDPQWMVRWNQFEOBPKSHNTTDOZIFCAJEHZGEPHA7PCFXG47MMSQQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.985602953 (493h7m47.856685988s) [noise=true , address=145.79.227.49:4337 , piece_id=NQ4MLAIAMOQX3LNYKZRAJBR62W2S3QE4LIIIRI6MWSEEWFMPW5UQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.985713162 (493h7m47.856796207s) [noise=true , address=145.79.227.41:4087 , piece_id=BHZPCHKZT62WR6DOCGXGY22OMZ7OTURG32FREETIVEVCOQKSKJ5Q, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.986095410 (493h7m47.857178445s) [noise=true , address=77.90.31.18:4805 , piece_id=LUHUR47JN4BQ54IJNUM6ZB54ZXERRJDLELGLXD44AIAMJDSPIHIQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.986472388 (493h7m47.857555433s) [noise=true , address=145.79.227.32:4835 , piece_id=SZYKTLLEJQGY6HQALZ6DQHDX3DUYUU35HTFH5XRYWK6OOWXNISVQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.986810162 (493h7m47.857893207s) [noise=true , address=145.79.227.45:4229 , piece_id=7QMCB2DGYPPJA4LEXBBQ4FTD2PNIFD42BCDYN3O2NY7SM4S4LIIQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.987775760 (493h7m47.858858795s) [noise=true , address=145.79.227.29:4943 , piece_id=ZBBDVYQWKPGFIOJNQQDEUKEWHDSBLVMZFIAXLB5D3A5URESWCCFA, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:49.126609907 (493h7m48.997692942s) [noise=true , address=77.90.31.18:4805 , piece_id=LUHUR47JN4BQ54IJNUM6ZB54ZXERRJDLELGLXD44AIAMJDSPIHIQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 04:36:49.694425487 (493h7m49.565508532s) [upload=432811]: Next returned nil segment Dec 26 04:36:49.694442139 (493h7m49.565525184s) [upload=432811]: Waiting for error group managing segments... Dec 26 04:36:49.979142989 (493h7m49.850226034s) [noise=true , address=145.79.227.19:4079 , piece_id=SCYS5EQ5U4XUS6KKPTOYTHRXJLDTBFRBDPRXYC7LOBYZNVWJOWYA, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:49.982012633 (493h7m49.853095678s) [noise=true , address=145.79.227.46:4602 , piece_id=ZNZ3ERQWYTJTNC3HVJ4D75BNGVWV7XBMLKROFANXU7YCJBWLIG4Q, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:49.984842081 (493h7m49.855925126s) [noise=true , address=145.79.227.52:4445 , piece_id=WDUBLLSJD7DZXF5RX4TKDSMCTOYJDK6YNEIYTS6PVKZZBU5LAZHQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:49.989501113 (493h7m49.860584148s) [noise=true , address=145.79.227.45:4229 , piece_id=7QMCB2DGYPPJA4LEXBBQ4FTD2PNIFD42BCDYN3O2NY7SM4S4LIIQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:49.996345335 (493h7m49.86742838s) [noise=true , address=145.79.227.36:4518 , piece_id=BLEKUBCF3JD6G5Z3BXBC6RZTNBUIHRZUSQZ7NYHZJN2TQXEQSVJA, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:49.996471736 (493h7m49.867554771s) [noise=true , address=145.79.227.32:4835 , piece_id=SZYKTLLEJQGY6HQALZ6DQHDX3DUYUU35HTFH5XRYWK6OOWXNISVQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.001016450 (493h7m49.872099505s) [noise=true , address=145.79.227.47:4266 , piece_id=6BHR4XPIWMPIVMQPX6LEZSG2V4VBUBILP2743K2TLGYFHLY5X5VA, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.010819717 (493h7m49.881902762s) [noise=true , address=145.79.227.41:4087 , piece_id=BHZPCHKZT62WR6DOCGXGY22OMZ7OTURG32FREETIVEVCOQKSKJ5Q, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.011633246 (493h7m49.882716291s) [noise=true , address=145.79.227.49:4337 , piece_id=NQ4MLAIAMOQX3LNYKZRAJBR62W2S3QE4LIIIRI6MWSEEWFMPW5UQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.040832877 (493h7m49.911915912s) [noise=true , address=145.79.227.43:4158 , piece_id=7CDPQWMVRWNQFEOBPKSHNTTDOZIFCAJEHZGEPHA7PCFXG47MMSQQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.040914794 (493h7m49.911997829s) [noise=true , address=145.79.227.29:4943 , piece_id=ZBBDVYQWKPGFIOJNQQDEUKEWHDSBLVMZFIAXLB5D3A5URESWCCFA, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.249402270 (493h7m50.120485315s) [noise=true , address=45.91.116.13:4154 , piece_id=6KNERLGIUZXTFJ6MV5GZOEVVLRCW3ODGIVVIAY5TAWSTNGJVLUZA, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.306072740 (493h7m50.177155775s) [noise=true , address=45.91.116.12:4226 , piece_id=PYNBIOFRYJ5LEUYUKQCY536NSEO2P6E5USVH6GVAE4XCSSNJRIDA, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.306117846 (493h7m50.177200881s) [seg_pos={0 0}, upload=432811]: Segment reached optimal threshold of 13 pieces. Dec 26 04:36:50.306300233 (493h7m50.177383278s) [noise=true , address=45.91.116.6:4974 , piece_id=LNIFEY4C5NIVJI5L2WHP7RLU2JHR7LLQEQB342U4EV2OPKLHE3HQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:36:50.306306495 (493h7m50.17738954s) [noise=true , address=45.91.116.6:4974 , piece_id=LNIFEY4C5NIVJI5L2WHP7RLU2JHR7LLQEQB342U4EV2OPKLHE3HQ, seg_pos={0 0}, upload=432811]: Long tail cancellation due to optimal threshold Dec 26 04:36:50.306515973 (493h7m50.177599008s) [noise=true , address=45.91.116.10:4084 , piece_id=7Y3VO42IM3G6TRT4LYVFGHOCRAV5TU3Z4HIV6LYYVYMGGYRUGCDQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:43178->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:43178->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43178->45.91.116.10:4084: use of closed network connection Dec 26 04:36:50.306534970 (493h7m50.177618015s) [noise=true , address=45.91.116.10:4084 , piece_id=7Y3VO42IM3G6TRT4LYVFGHOCRAV5TU3Z4HIV6LYYVYMGGYRUGCDQ, seg_pos={0 0}, upload=432811]: Long tail cancellation due to optimal threshold Dec 26 04:36:50.306576609 (493h7m50.177659654s) [seg_pos={0 0}, upload=432811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:36:50.306693663 (493h7m50.177776708s) [upload=432811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:36:50.306699604 (493h7m50.177782649s) [upload=432811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:36:50.602447198 (493h7m50.473530233s) [upload=432811]: Done uploading segments... Dec 26 04:36:50.602466445 (493h7m50.473549481s) [upload=432811]: Upload finished. err: <nil> Duration: 5.700927646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 33.676658263s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:37:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 33.676697799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.155659205s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:37:18 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.155686857s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................................X..X..X.X....XX.X.X] duration: 30.002623616s trace: https://jaeger.dev.storj.io/trace/6be8f72b02990c84 FAIL (2025-12-26 04:37:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 04:36:47.735473833 (493h7m47.606556868s) [upload=432811]: Starting upload Dec 26 04:36:47.735529369 (493h7m47.606612404s) [upload=432811]: Done starting upload Dec 26 04:36:47.735604121 (493h7m47.606687156s) [upload=432811]: Uploading segments... Dec 26 04:36:47.735835983 (493h7m47.606919018s) [upload=432811]: Got next segment. Inline: false Dec 26 04:36:47.735866471 (493h7m47.606949506s) [upload=432811]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:36:47.735871361 (493h7m47.606954396s) [upload=432811]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:36:47.983428432 (493h7m47.854511468s) [seg_pos={0 0}, upload=432811]: Begin upload segment... Dec 26 04:36:47.983474320 (493h7m47.854557365s) [seg_pos={0 0}, upload=432811]: Done begin upload segment. Dec 26 04:36:47.983500750 (493h7m47.854583795s) [noise=true , address=145.79.227.36:4518 , piece_id=BLEKUBCF3JD6G5Z3BXBC6RZTNBUIHRZUSQZ7NYHZJN2TQXEQSVJA, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.983515428 (493h7m47.854598463s) [noise=true , address=145.79.227.47:4266 , piece_id=6BHR4XPIWMPIVMQPX6LEZSG2V4VBUBILP2743K2TLGYFHLY5X5VA, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.983598146 (493h7m47.854681181s) [noise=true , address=145.79.227.46:4602 , piece_id=ZNZ3ERQWYTJTNC3HVJ4D75BNGVWV7XBMLKROFANXU7YCJBWLIG4Q, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.984082268 (493h7m47.855165313s) [noise=true , address=145.79.227.52:4445 , piece_id=WDUBLLSJD7DZXF5RX4TKDSMCTOYJDK6YNEIYTS6PVKZZBU5LAZHQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.984402117 (493h7m47.855485152s) [noise=true , address=45.91.116.6:4974 , piece_id=LNIFEY4C5NIVJI5L2WHP7RLU2JHR7LLQEQB342U4EV2OPKLHE3HQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.984702778 (493h7m47.855785813s) [noise=true , address=45.91.116.10:4084 , piece_id=7Y3VO42IM3G6TRT4LYVFGHOCRAV5TU3Z4HIV6LYYVYMGGYRUGCDQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.985048998 (493h7m47.856132043s) [noise=true , address=145.79.227.19:4079 , piece_id=SCYS5EQ5U4XUS6KKPTOYTHRXJLDTBFRBDPRXYC7LOBYZNVWJOWYA, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.985056593 (493h7m47.856139628s) [noise=true , address=45.91.116.13:4154 , piece_id=6KNERLGIUZXTFJ6MV5GZOEVVLRCW3ODGIVVIAY5TAWSTNGJVLUZA, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.985330894 (493h7m47.856413929s) [noise=true , address=45.91.116.12:4226 , piece_id=PYNBIOFRYJ5LEUYUKQCY536NSEO2P6E5USVH6GVAE4XCSSNJRIDA, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.985425695 (493h7m47.85650873s) [noise=true , address=145.79.227.43:4158 , piece_id=7CDPQWMVRWNQFEOBPKSHNTTDOZIFCAJEHZGEPHA7PCFXG47MMSQQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.985602953 (493h7m47.856685988s) [noise=true , address=145.79.227.49:4337 , piece_id=NQ4MLAIAMOQX3LNYKZRAJBR62W2S3QE4LIIIRI6MWSEEWFMPW5UQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.985713162 (493h7m47.856796207s) [noise=true , address=145.79.227.41:4087 , piece_id=BHZPCHKZT62WR6DOCGXGY22OMZ7OTURG32FREETIVEVCOQKSKJ5Q, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.986095410 (493h7m47.857178445s) [noise=true , address=77.90.31.18:4805 , piece_id=LUHUR47JN4BQ54IJNUM6ZB54ZXERRJDLELGLXD44AIAMJDSPIHIQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.986472388 (493h7m47.857555433s) [noise=true , address=145.79.227.32:4835 , piece_id=SZYKTLLEJQGY6HQALZ6DQHDX3DUYUU35HTFH5XRYWK6OOWXNISVQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.986810162 (493h7m47.857893207s) [noise=true , address=145.79.227.45:4229 , piece_id=7QMCB2DGYPPJA4LEXBBQ4FTD2PNIFD42BCDYN3O2NY7SM4S4LIIQ, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:47.987775760 (493h7m47.858858795s) [noise=true , address=145.79.227.29:4943 , piece_id=ZBBDVYQWKPGFIOJNQQDEUKEWHDSBLVMZFIAXLB5D3A5URESWCCFA, seg_pos={0 0}, upload=432811]: Uploading piece... Dec 26 04:36:49.126609907 (493h7m48.997692942s) [noise=true , address=77.90.31.18:4805 , piece_id=LUHUR47JN4BQ54IJNUM6ZB54ZXERRJDLELGLXD44AIAMJDSPIHIQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 04:36:49.694425487 (493h7m49.565508532s) [upload=432811]: Next returned nil segment Dec 26 04:36:49.694442139 (493h7m49.565525184s) [upload=432811]: Waiting for error group managing segments... Dec 26 04:36:49.979142989 (493h7m49.850226034s) [noise=true , address=145.79.227.19:4079 , piece_id=SCYS5EQ5U4XUS6KKPTOYTHRXJLDTBFRBDPRXYC7LOBYZNVWJOWYA, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:49.982012633 (493h7m49.853095678s) [noise=true , address=145.79.227.46:4602 , piece_id=ZNZ3ERQWYTJTNC3HVJ4D75BNGVWV7XBMLKROFANXU7YCJBWLIG4Q, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:49.984842081 (493h7m49.855925126s) [noise=true , address=145.79.227.52:4445 , piece_id=WDUBLLSJD7DZXF5RX4TKDSMCTOYJDK6YNEIYTS6PVKZZBU5LAZHQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:49.989501113 (493h7m49.860584148s) [noise=true , address=145.79.227.45:4229 , piece_id=7QMCB2DGYPPJA4LEXBBQ4FTD2PNIFD42BCDYN3O2NY7SM4S4LIIQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:49.996345335 (493h7m49.86742838s) [noise=true , address=145.79.227.36:4518 , piece_id=BLEKUBCF3JD6G5Z3BXBC6RZTNBUIHRZUSQZ7NYHZJN2TQXEQSVJA, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:49.996471736 (493h7m49.867554771s) [noise=true , address=145.79.227.32:4835 , piece_id=SZYKTLLEJQGY6HQALZ6DQHDX3DUYUU35HTFH5XRYWK6OOWXNISVQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.001016450 (493h7m49.872099505s) [noise=true , address=145.79.227.47:4266 , piece_id=6BHR4XPIWMPIVMQPX6LEZSG2V4VBUBILP2743K2TLGYFHLY5X5VA, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.010819717 (493h7m49.881902762s) [noise=true , address=145.79.227.41:4087 , piece_id=BHZPCHKZT62WR6DOCGXGY22OMZ7OTURG32FREETIVEVCOQKSKJ5Q, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.011633246 (493h7m49.882716291s) [noise=true , address=145.79.227.49:4337 , piece_id=NQ4MLAIAMOQX3LNYKZRAJBR62W2S3QE4LIIIRI6MWSEEWFMPW5UQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.040832877 (493h7m49.911915912s) [noise=true , address=145.79.227.43:4158 , piece_id=7CDPQWMVRWNQFEOBPKSHNTTDOZIFCAJEHZGEPHA7PCFXG47MMSQQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.040914794 (493h7m49.911997829s) [noise=true , address=145.79.227.29:4943 , piece_id=ZBBDVYQWKPGFIOJNQQDEUKEWHDSBLVMZFIAXLB5D3A5URESWCCFA, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.249402270 (493h7m50.120485315s) [noise=true , address=45.91.116.13:4154 , piece_id=6KNERLGIUZXTFJ6MV5GZOEVVLRCW3ODGIVVIAY5TAWSTNGJVLUZA, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.306072740 (493h7m50.177155775s) [noise=true , address=45.91.116.12:4226 , piece_id=PYNBIOFRYJ5LEUYUKQCY536NSEO2P6E5USVH6GVAE4XCSSNJRIDA, seg_pos={0 0}, upload=432811]: Done uploading piece. err: <nil> Dec 26 04:36:50.306117846 (493h7m50.177200881s) [seg_pos={0 0}, upload=432811]: Segment reached optimal threshold of 13 pieces. Dec 26 04:36:50.306300233 (493h7m50.177383278s) [noise=true , address=45.91.116.6:4974 , piece_id=LNIFEY4C5NIVJI5L2WHP7RLU2JHR7LLQEQB342U4EV2OPKLHE3HQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:36:50.306306495 (493h7m50.17738954s) [noise=true , address=45.91.116.6:4974 , piece_id=LNIFEY4C5NIVJI5L2WHP7RLU2JHR7LLQEQB342U4EV2OPKLHE3HQ, seg_pos={0 0}, upload=432811]: Long tail cancellation due to optimal threshold Dec 26 04:36:50.306515973 (493h7m50.177599008s) [noise=true , address=45.91.116.10:4084 , piece_id=7Y3VO42IM3G6TRT4LYVFGHOCRAV5TU3Z4HIV6LYYVYMGGYRUGCDQ, seg_pos={0 0}, upload=432811]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:43178->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:43178->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43178->45.91.116.10:4084: use of closed network connection Dec 26 04:36:50.306534970 (493h7m50.177618015s) [noise=true , address=45.91.116.10:4084 , piece_id=7Y3VO42IM3G6TRT4LYVFGHOCRAV5TU3Z4HIV6LYYVYMGGYRUGCDQ, seg_pos={0 0}, upload=432811]: Long tail cancellation due to optimal threshold Dec 26 04:36:50.306576609 (493h7m50.177659654s) [seg_pos={0 0}, upload=432811]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:36:50.306693663 (493h7m50.177776708s) [upload=432811]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:36:50.306699604 (493h7m50.177782649s) [upload=432811]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:36:50.602447198 (493h7m50.473530233s) [upload=432811]: Done uploading segments... Dec 26 04:36:50.602466445 (493h7m50.473549481s) [upload=432811]: Upload finished. err: <nil> Duration: 30.002642121s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 30.091807016s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:36:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 30.091835722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................................X..X..X.X....XX.X.] duration: 6.213048974s trace: https://jaeger.dev.storj.io/trace/5ffe037cc8e072cb FAIL (2025-12-26 04:36:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 04:35:22.411027152 (493h6m22.282110187s) [upload=432791]: Starting upload Dec 26 04:35:22.411100311 (493h6m22.282183377s) [upload=432791]: Done starting upload Dec 26 04:35:22.411250086 (493h6m22.282333121s) [upload=432791]: Uploading segments... Dec 26 04:35:22.411329988 (493h6m22.282413033s) [upload=432791]: Got next segment. Inline: false Dec 26 04:35:22.411363171 (493h6m22.282446206s) [upload=432791]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:35:22.411368301 (493h6m22.282451336s) [upload=432791]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:35:22.656599895 (493h6m22.52768295s) [seg_pos={0 0}, upload=432791]: Begin upload segment... Dec 26 04:35:22.656662073 (493h6m22.527745118s) [seg_pos={0 0}, upload=432791]: Done begin upload segment. Dec 26 04:35:22.656719974 (493h6m22.527803029s) [noise=true , address=77.90.31.10:4519 , piece_id=DIAE2SY4FZ2DVIVHRMLFKZTFUI3WGS55HV3WVLB72YAPAKT6CWZQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.656730844 (493h6m22.527813879s) [noise=true , address=77.90.31.11:4555 , piece_id=HALDVQJWATPDVB3GQXWRKM5YXERTANNVBUFQRLXWKGVK5V4CHYTQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.657679009 (493h6m22.528762055s) [noise=true , address=145.79.227.14:4837 , piece_id=3YL6OL3IISMM4UG3PO62YTPWSUGJCF43EIAMJ3R62SGSS5W4TOSA, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.658183691 (493h6m22.529266736s) [noise=true , address=77.90.31.18:4808 , piece_id=WUEZGCWYTRIJ55HTPSG6OI43QFCXTXBFSFM2AVRUFYNHGJS63AIQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.658768254 (493h6m22.529851299s) [noise=true , address=145.79.227.42:4123 , piece_id=SILNWZ4IN5AK2S7YM7HVYDUZS5BRPXX3WCEX4YSTBXQURSKGVKUA, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.659275670 (493h6m22.530358715s) [noise=true , address=45.91.116.10:4084 , piece_id=HEFJDKITF3ZXNLJTSU2TAZX5L6CQ4SXCVQXTKDSXVZSZJEODT6AQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.659742770 (493h6m22.530825815s) [noise=true , address=145.79.227.8:4329 , piece_id=4DQZ4EMNM32XTD7Q6TZAIINGLW4E663PF3XPMJPX3VDQ2RWAQWYQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.660275835 (493h6m22.53135888s) [noise=true , address=145.79.227.34:4906 , piece_id=JELENGAA4EOMMOMNDASUMWN4K5O72ACARS5Y7PWQVZIRRGVPG3CQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.661694096 (493h6m22.532777131s) [noise=true , address=145.79.227.35:4483 , piece_id=G6PVNYHIK7UMABHMJDX5GKSRTQX366T3BZCG6MWH7YCNABK4IDHQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.662005669 (493h6m22.533088714s) [noise=true , address=145.79.227.39:4013 , piece_id=757C3ZAHA3OIQLJ55UVO732B24ULHRC5CDRQGD7K4LNLBYOTOQBA, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.663765369 (493h6m22.534848404s) [noise=true , address=45.91.116.4:4937 , piece_id=QIHNHVA4AIKDDTBYHXLXCO7GD4B2MOLSXACQV5ZYBQ2N3LR77JHQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.664305498 (493h6m22.535388563s) [noise=true , address=145.79.227.45:4231 , piece_id=AJXL4HDRQX7NNUG2VPS2QLW4TIQWCKUN6HTGQYP5QLGV7D42QJZA, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.665403929 (493h6m22.536486974s) [noise=true , address=45.91.116.12:4226 , piece_id=AG5EDRATLOA6VYUFMDJDEQUW6SGUIUID4JINU3E7ITIWEAAPUHYA, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.666228970 (493h6m22.537312026s) [noise=true , address=145.79.227.11:4731 , piece_id=WDNFIS556OUVMVQKG4ZYTJYEMRCV7DK44OPCXBOHCPCSJOJ6YDMA, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.667503668 (493h6m22.538586713s) [noise=true , address=45.91.116.13:4154 , piece_id=TN44MA34XFZC6A6L5DGR7ZPKOXWWYVZ4WGEJELNRHHJCJXU4SMGQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.667987710 (493h6m22.539070765s) [noise=true , address=45.91.116.6:4974 , piece_id=QNI5E6AJYNZIOGPXAEMUQDGGRMGQCX2UUHORTCIZJGF4DH5LXU4A, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:24.397988611 (493h6m24.269071646s) [upload=432791]: Next returned nil segment Dec 26 04:35:24.398003099 (493h6m24.269086144s) [upload=432791]: Waiting for error group managing segments... Dec 26 04:35:24.679463217 (493h6m24.550546262s) [noise=true , address=145.79.227.14:4837 , piece_id=3YL6OL3IISMM4UG3PO62YTPWSUGJCF43EIAMJ3R62SGSS5W4TOSA, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.680736202 (493h6m24.551819237s) [noise=true , address=145.79.227.8:4329 , piece_id=4DQZ4EMNM32XTD7Q6TZAIINGLW4E663PF3XPMJPX3VDQ2RWAQWYQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.681241974 (493h6m24.552325019s) [noise=true , address=145.79.227.45:4231 , piece_id=AJXL4HDRQX7NNUG2VPS2QLW4TIQWCKUN6HTGQYP5QLGV7D42QJZA, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.692726924 (493h6m24.563809969s) [noise=true , address=145.79.227.39:4013 , piece_id=757C3ZAHA3OIQLJ55UVO732B24ULHRC5CDRQGD7K4LNLBYOTOQBA, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.693592602 (493h6m24.564675637s) [noise=true , address=145.79.227.11:4731 , piece_id=WDNFIS556OUVMVQKG4ZYTJYEMRCV7DK44OPCXBOHCPCSJOJ6YDMA, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.696541317 (493h6m24.567624352s) [noise=true , address=145.79.227.42:4123 , piece_id=SILNWZ4IN5AK2S7YM7HVYDUZS5BRPXX3WCEX4YSTBXQURSKGVKUA, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.706680474 (493h6m24.577763509s) [noise=true , address=145.79.227.35:4483 , piece_id=G6PVNYHIK7UMABHMJDX5GKSRTQX366T3BZCG6MWH7YCNABK4IDHQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.958229588 (493h6m24.829312623s) [noise=true , address=77.90.31.11:4555 , piece_id=HALDVQJWATPDVB3GQXWRKM5YXERTANNVBUFQRLXWKGVK5V4CHYTQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.976417159 (493h6m24.847500194s) [noise=true , address=45.91.116.6:4974 , piece_id=QNI5E6AJYNZIOGPXAEMUQDGGRMGQCX2UUHORTCIZJGF4DH5LXU4A, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:25.017941289 (493h6m24.889024374s) [noise=true , address=77.90.31.10:4519 , piece_id=DIAE2SY4FZ2DVIVHRMLFKZTFUI3WGS55HV3WVLB72YAPAKT6CWZQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:25.021398222 (493h6m24.892481257s) [noise=true , address=45.91.116.12:4226 , piece_id=AG5EDRATLOA6VYUFMDJDEQUW6SGUIUID4JINU3E7ITIWEAAPUHYA, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:25.226530478 (493h6m25.097613523s) [noise=true , address=45.91.116.13:4154 , piece_id=TN44MA34XFZC6A6L5DGR7ZPKOXWWYVZ4WGEJELNRHHJCJXU4SMGQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:25.602982409 (493h6m25.474065454s) [noise=true , address=77.90.31.18:4808 , piece_id=WUEZGCWYTRIJ55HTPSG6OI43QFCXTXBFSFM2AVRUFYNHGJS63AIQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:25.603008407 (493h6m25.474091442s) [seg_pos={0 0}, upload=432791]: Segment reached optimal threshold of 13 pieces. Dec 26 04:35:25.603255888 (493h6m25.474338933s) [noise=true , address=45.91.116.4:4937 , piece_id=QIHNHVA4AIKDDTBYHXLXCO7GD4B2MOLSXACQV5ZYBQ2N3LR77JHQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 04:35:25.603266439 (493h6m25.474349484s) [noise=true , address=45.91.116.4:4937 , piece_id=QIHNHVA4AIKDDTBYHXLXCO7GD4B2MOLSXACQV5ZYBQ2N3LR77JHQ, seg_pos={0 0}, upload=432791]: Long tail cancellation due to optimal threshold Dec 26 04:35:25.603432024 (493h6m25.474515069s) [noise=true , address=45.91.116.10:4084 , piece_id=HEFJDKITF3ZXNLJTSU2TAZX5L6CQ4SXCVQXTKDSXVZSZJEODT6AQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:34444->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:34444->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34444->45.91.116.10:4084: use of closed network connection Dec 26 04:35:25.603440210 (493h6m25.474523255s) [noise=true , address=45.91.116.10:4084 , piece_id=HEFJDKITF3ZXNLJTSU2TAZX5L6CQ4SXCVQXTKDSXVZSZJEODT6AQ, seg_pos={0 0}, upload=432791]: Long tail cancellation due to optimal threshold Dec 26 04:35:25.603742585 (493h6m25.47482564s) [noise=true , address=145.79.227.34:4906 , piece_id=JELENGAA4EOMMOMNDASUMWN4K5O72ACARS5Y7PWQVZIRRGVPG3CQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 04:35:25.603762754 (493h6m25.474845789s) [noise=true , address=145.79.227.34:4906 , piece_id=JELENGAA4EOMMOMNDASUMWN4K5O72ACARS5Y7PWQVZIRRGVPG3CQ, seg_pos={0 0}, upload=432791]: Long tail cancellation due to optimal threshold Dec 26 04:35:25.603792080 (493h6m25.474875125s) [seg_pos={0 0}, upload=432791]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:35:25.603949980 (493h6m25.475033025s) [upload=432791]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:35:25.603958958 (493h6m25.475042003s) [upload=432791]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:35:25.895054965 (493h6m25.766138s) [upload=432791]: Done uploading segments... Dec 26 04:35:25.895079331 (493h6m25.766162367s) [upload=432791]: Upload finished. err: <nil> Duration: 6.213081837s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.25531706s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:36:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.255368097s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 41.093230096s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:35:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 41.093284208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................................X..X..X.X....XX.X] duration: 30.001891149s trace: https://jaeger.dev.storj.io/trace/5ffe037cc8e072cb FAIL (2025-12-26 04:35:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 04:35:22.411027152 (493h6m22.282110187s) [upload=432791]: Starting upload Dec 26 04:35:22.411100311 (493h6m22.282183377s) [upload=432791]: Done starting upload Dec 26 04:35:22.411250086 (493h6m22.282333121s) [upload=432791]: Uploading segments... Dec 26 04:35:22.411329988 (493h6m22.282413033s) [upload=432791]: Got next segment. Inline: false Dec 26 04:35:22.411363171 (493h6m22.282446206s) [upload=432791]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:35:22.411368301 (493h6m22.282451336s) [upload=432791]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:35:22.656599895 (493h6m22.52768295s) [seg_pos={0 0}, upload=432791]: Begin upload segment... Dec 26 04:35:22.656662073 (493h6m22.527745118s) [seg_pos={0 0}, upload=432791]: Done begin upload segment. Dec 26 04:35:22.656719974 (493h6m22.527803029s) [noise=true , address=77.90.31.10:4519 , piece_id=DIAE2SY4FZ2DVIVHRMLFKZTFUI3WGS55HV3WVLB72YAPAKT6CWZQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.656730844 (493h6m22.527813879s) [noise=true , address=77.90.31.11:4555 , piece_id=HALDVQJWATPDVB3GQXWRKM5YXERTANNVBUFQRLXWKGVK5V4CHYTQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.657679009 (493h6m22.528762055s) [noise=true , address=145.79.227.14:4837 , piece_id=3YL6OL3IISMM4UG3PO62YTPWSUGJCF43EIAMJ3R62SGSS5W4TOSA, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.658183691 (493h6m22.529266736s) [noise=true , address=77.90.31.18:4808 , piece_id=WUEZGCWYTRIJ55HTPSG6OI43QFCXTXBFSFM2AVRUFYNHGJS63AIQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.658768254 (493h6m22.529851299s) [noise=true , address=145.79.227.42:4123 , piece_id=SILNWZ4IN5AK2S7YM7HVYDUZS5BRPXX3WCEX4YSTBXQURSKGVKUA, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.659275670 (493h6m22.530358715s) [noise=true , address=45.91.116.10:4084 , piece_id=HEFJDKITF3ZXNLJTSU2TAZX5L6CQ4SXCVQXTKDSXVZSZJEODT6AQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.659742770 (493h6m22.530825815s) [noise=true , address=145.79.227.8:4329 , piece_id=4DQZ4EMNM32XTD7Q6TZAIINGLW4E663PF3XPMJPX3VDQ2RWAQWYQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.660275835 (493h6m22.53135888s) [noise=true , address=145.79.227.34:4906 , piece_id=JELENGAA4EOMMOMNDASUMWN4K5O72ACARS5Y7PWQVZIRRGVPG3CQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.661694096 (493h6m22.532777131s) [noise=true , address=145.79.227.35:4483 , piece_id=G6PVNYHIK7UMABHMJDX5GKSRTQX366T3BZCG6MWH7YCNABK4IDHQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.662005669 (493h6m22.533088714s) [noise=true , address=145.79.227.39:4013 , piece_id=757C3ZAHA3OIQLJ55UVO732B24ULHRC5CDRQGD7K4LNLBYOTOQBA, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.663765369 (493h6m22.534848404s) [noise=true , address=45.91.116.4:4937 , piece_id=QIHNHVA4AIKDDTBYHXLXCO7GD4B2MOLSXACQV5ZYBQ2N3LR77JHQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.664305498 (493h6m22.535388563s) [noise=true , address=145.79.227.45:4231 , piece_id=AJXL4HDRQX7NNUG2VPS2QLW4TIQWCKUN6HTGQYP5QLGV7D42QJZA, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.665403929 (493h6m22.536486974s) [noise=true , address=45.91.116.12:4226 , piece_id=AG5EDRATLOA6VYUFMDJDEQUW6SGUIUID4JINU3E7ITIWEAAPUHYA, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.666228970 (493h6m22.537312026s) [noise=true , address=145.79.227.11:4731 , piece_id=WDNFIS556OUVMVQKG4ZYTJYEMRCV7DK44OPCXBOHCPCSJOJ6YDMA, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.667503668 (493h6m22.538586713s) [noise=true , address=45.91.116.13:4154 , piece_id=TN44MA34XFZC6A6L5DGR7ZPKOXWWYVZ4WGEJELNRHHJCJXU4SMGQ, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:22.667987710 (493h6m22.539070765s) [noise=true , address=45.91.116.6:4974 , piece_id=QNI5E6AJYNZIOGPXAEMUQDGGRMGQCX2UUHORTCIZJGF4DH5LXU4A, seg_pos={0 0}, upload=432791]: Uploading piece... Dec 26 04:35:24.397988611 (493h6m24.269071646s) [upload=432791]: Next returned nil segment Dec 26 04:35:24.398003099 (493h6m24.269086144s) [upload=432791]: Waiting for error group managing segments... Dec 26 04:35:24.679463217 (493h6m24.550546262s) [noise=true , address=145.79.227.14:4837 , piece_id=3YL6OL3IISMM4UG3PO62YTPWSUGJCF43EIAMJ3R62SGSS5W4TOSA, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.680736202 (493h6m24.551819237s) [noise=true , address=145.79.227.8:4329 , piece_id=4DQZ4EMNM32XTD7Q6TZAIINGLW4E663PF3XPMJPX3VDQ2RWAQWYQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.681241974 (493h6m24.552325019s) [noise=true , address=145.79.227.45:4231 , piece_id=AJXL4HDRQX7NNUG2VPS2QLW4TIQWCKUN6HTGQYP5QLGV7D42QJZA, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.692726924 (493h6m24.563809969s) [noise=true , address=145.79.227.39:4013 , piece_id=757C3ZAHA3OIQLJ55UVO732B24ULHRC5CDRQGD7K4LNLBYOTOQBA, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.693592602 (493h6m24.564675637s) [noise=true , address=145.79.227.11:4731 , piece_id=WDNFIS556OUVMVQKG4ZYTJYEMRCV7DK44OPCXBOHCPCSJOJ6YDMA, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.696541317 (493h6m24.567624352s) [noise=true , address=145.79.227.42:4123 , piece_id=SILNWZ4IN5AK2S7YM7HVYDUZS5BRPXX3WCEX4YSTBXQURSKGVKUA, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.706680474 (493h6m24.577763509s) [noise=true , address=145.79.227.35:4483 , piece_id=G6PVNYHIK7UMABHMJDX5GKSRTQX366T3BZCG6MWH7YCNABK4IDHQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.958229588 (493h6m24.829312623s) [noise=true , address=77.90.31.11:4555 , piece_id=HALDVQJWATPDVB3GQXWRKM5YXERTANNVBUFQRLXWKGVK5V4CHYTQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:24.976417159 (493h6m24.847500194s) [noise=true , address=45.91.116.6:4974 , piece_id=QNI5E6AJYNZIOGPXAEMUQDGGRMGQCX2UUHORTCIZJGF4DH5LXU4A, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:25.017941289 (493h6m24.889024374s) [noise=true , address=77.90.31.10:4519 , piece_id=DIAE2SY4FZ2DVIVHRMLFKZTFUI3WGS55HV3WVLB72YAPAKT6CWZQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:25.021398222 (493h6m24.892481257s) [noise=true , address=45.91.116.12:4226 , piece_id=AG5EDRATLOA6VYUFMDJDEQUW6SGUIUID4JINU3E7ITIWEAAPUHYA, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:25.226530478 (493h6m25.097613523s) [noise=true , address=45.91.116.13:4154 , piece_id=TN44MA34XFZC6A6L5DGR7ZPKOXWWYVZ4WGEJELNRHHJCJXU4SMGQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:25.602982409 (493h6m25.474065454s) [noise=true , address=77.90.31.18:4808 , piece_id=WUEZGCWYTRIJ55HTPSG6OI43QFCXTXBFSFM2AVRUFYNHGJS63AIQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: <nil> Dec 26 04:35:25.603008407 (493h6m25.474091442s) [seg_pos={0 0}, upload=432791]: Segment reached optimal threshold of 13 pieces. Dec 26 04:35:25.603255888 (493h6m25.474338933s) [noise=true , address=45.91.116.4:4937 , piece_id=QIHNHVA4AIKDDTBYHXLXCO7GD4B2MOLSXACQV5ZYBQ2N3LR77JHQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 04:35:25.603266439 (493h6m25.474349484s) [noise=true , address=45.91.116.4:4937 , piece_id=QIHNHVA4AIKDDTBYHXLXCO7GD4B2MOLSXACQV5ZYBQ2N3LR77JHQ, seg_pos={0 0}, upload=432791]: Long tail cancellation due to optimal threshold Dec 26 04:35:25.603432024 (493h6m25.474515069s) [noise=true , address=45.91.116.10:4084 , piece_id=HEFJDKITF3ZXNLJTSU2TAZX5L6CQ4SXCVQXTKDSXVZSZJEODT6AQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:34444->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:34444->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34444->45.91.116.10:4084: use of closed network connection Dec 26 04:35:25.603440210 (493h6m25.474523255s) [noise=true , address=45.91.116.10:4084 , piece_id=HEFJDKITF3ZXNLJTSU2TAZX5L6CQ4SXCVQXTKDSXVZSZJEODT6AQ, seg_pos={0 0}, upload=432791]: Long tail cancellation due to optimal threshold Dec 26 04:35:25.603742585 (493h6m25.47482564s) [noise=true , address=145.79.227.34:4906 , piece_id=JELENGAA4EOMMOMNDASUMWN4K5O72ACARS5Y7PWQVZIRRGVPG3CQ, seg_pos={0 0}, upload=432791]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 04:35:25.603762754 (493h6m25.474845789s) [noise=true , address=145.79.227.34:4906 , piece_id=JELENGAA4EOMMOMNDASUMWN4K5O72ACARS5Y7PWQVZIRRGVPG3CQ, seg_pos={0 0}, upload=432791]: Long tail cancellation due to optimal threshold Dec 26 04:35:25.603792080 (493h6m25.474875125s) [seg_pos={0 0}, upload=432791]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:35:25.603949980 (493h6m25.475033025s) [upload=432791]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:35:25.603958958 (493h6m25.475042003s) [upload=432791]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:35:25.895054965 (493h6m25.766138s) [upload=432791]: Done uploading segments... Dec 26 04:35:25.895079331 (493h6m25.766162367s) [upload=432791]: Upload finished. err: <nil> Duration: 30.001925735s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.623951413s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:35:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.62397591s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.066943084s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:35:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.066973312s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 36.688865793s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:34:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 36.688893346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................................X..X..X.X....XX.] duration: 15.985730037s trace: https://jaeger.dev.storj.io/trace/21ffd64f6429782 FAIL (2025-12-26 04:34:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: EME54IOT2RMJM3CGSXGY4VRBUTDIBSD4PHGYDYDG7BGSQUHKENRA) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: HVMMW6PU2JR7P53RCYRG26WGHX4LDYETAKU5FP3FKYGEFDHQBGGA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: MLANYW6PPT4RXIIV6CWZJEQNC6DHFSENPPNPWOYYCYF7LEBXL2IA) context deadline exceeded ======LOGS====== Dec 26 04:33:47.666005705 (493h4m47.53708874s) [upload=432766]: Starting upload Dec 26 04:33:47.666076099 (493h4m47.537159164s) [upload=432766]: Done starting upload Dec 26 04:33:47.666297852 (493h4m47.537380897s) [upload=432766]: Uploading segments... Dec 26 04:33:47.666409384 (493h4m47.537492429s) [upload=432766]: Got next segment. Inline: false Dec 26 04:33:47.666434823 (493h4m47.537517868s) [upload=432766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:33:47.666441485 (493h4m47.53752452s) [upload=432766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:33:47.907599031 (493h4m47.778682106s) [seg_pos={0 0}, upload=432766]: Begin upload segment... Dec 26 04:33:47.907652533 (493h4m47.778735568s) [seg_pos={0 0}, upload=432766]: Done begin upload segment. Dec 26 04:33:47.907682350 (493h4m47.778765385s) [noise=true , address=77.90.31.13:4627 , piece_id=AGEDJ42SFMJGNQR6IHWYJ7564VJVPTDVJ4BVKLBS3SYBEGG3262Q, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.907696777 (493h4m47.778779822s) [noise=true , address=77.90.31.11:4555 , piece_id=MLANYW6PPT4RXIIV6CWZJEQNC6DHFSENPPNPWOYYCYF7LEBXL2IA, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.907745601 (493h4m47.778828636s) [noise=true , address=145.79.227.36:4517 , piece_id=EP4MV6YHY6NMPDSNCMOEIFYKEJEZEMOGNZ45MP33EYN3QHOXJTEQ, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.908233830 (493h4m47.779316876s) [noise=true , address=145.79.227.15:4876 , piece_id=MGPE2NJVQUOSTFVVBTOSYXNV6ZF67YRKWF2TS4KQBV4G5Y6EUC3Q, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.908874089 (493h4m47.779957134s) [noise=true , address=77.90.31.10:4519 , piece_id=ZC54VVALPOY3IH4OQPUU6DPYFABV63SV33KV45ODP3WUDAOHTIYA, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.908882936 (493h4m47.779965981s) [noise=true , address=77.90.31.19:4844 , piece_id=AZONQAM7M7ARQFIBFQKUDWU464G3PSNYMHXFDOBAJU36YAHWDVFA, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.909180563 (493h4m47.780263598s) [noise=true , address=145.79.227.35:4482 , piece_id=EME54IOT2RMJM3CGSXGY4VRBUTDIBSD4PHGYDYDG7BGSQUHKENRA, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.909441690 (493h4m47.780524735s) [noise=true , address=145.79.227.49:4338 , piece_id=MMJ5CQ7O6NQGBN7PJ4DOPJCPDXYZVDPATNUY4244GBQSYNLEOOGQ, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.909517794 (493h4m47.78060084s) [noise=true , address=145.79.227.47:4266 , piece_id=O4PKILRO3FBSQUVTU3FLL57FYLZOMIYXEEDZMOAZ7UFM7IEJ3T7Q, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.909893009 (493h4m47.780976054s) [noise=true , address=145.79.227.50:4373 , piece_id=EUSFDA3KK3QRC727PJ2ZBU3OXX7FBJEZ2O4EC2NP52LUXBLGYT5A, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.910006015 (493h4m47.78108906s) [noise=true , address=145.79.227.46:4602 , piece_id=E362APFCNUVQRYRNIKABYFY2NRZZ7WY6GGM5PBBORR3OCUQQ3MPQ, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.910182020 (493h4m47.781265065s) [noise=true , address=145.79.227.44:4195 , piece_id=4IWWRAPTBN7MSSTAABZ4R64MTOMXXOSFL4TQQDTK2RPV6LQIHH5A, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.910668937 (493h4m47.781751982s) [noise=true , address=145.79.227.6:4257 , piece_id=DFQDMCW5ONXE3W6QHSQCNQD6SJQPGC7Q643HVE6WBEIPQG64JJFQ, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.910684847 (493h4m47.781767892s) [noise=true , address=45.91.116.11:4119 , piece_id=XP7753K4QWBLQTE56O4OUL5R7TRKGM2HL667ENUUWKZVOOKK4DWQ, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.911193266 (493h4m47.782276311s) [noise=true , address=145.79.227.37:4567 , piece_id=HVMMW6PU2JR7P53RCYRG26WGHX4LDYETAKU5FP3FKYGEFDHQBGGA, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.913028299 (493h4m47.784111344s) [noise=true , address=145.79.227.19:4079 , piece_id=ZEOFO5JGYYT76IGCHSQY47OR4TFSBAS7SC33QDS44H7RJXKAZZ5Q, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:49.771832237 (493h4m49.642915282s) [upload=432766]: Next returned nil segment Dec 26 04:33:49.771848548 (493h4m49.642931613s) [upload=432766]: Waiting for error group managing segments... Dec 26 04:33:50.163660729 (493h4m50.034743775s) [noise=true , address=77.90.31.19:4844 , piece_id=AZONQAM7M7ARQFIBFQKUDWU464G3PSNYMHXFDOBAJU36YAHWDVFA, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:50.244521202 (493h4m50.115604247s) [noise=true , address=45.91.116.11:4119 , piece_id=XP7753K4QWBLQTE56O4OUL5R7TRKGM2HL667ENUUWKZVOOKK4DWQ, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:50.307098661 (493h4m50.178181716s) [noise=true , address=77.90.31.13:4627 , piece_id=AGEDJ42SFMJGNQR6IHWYJ7564VJVPTDVJ4BVKLBS3SYBEGG3262Q, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:50.416730701 (493h4m50.287813736s) [noise=true , address=77.90.31.10:4519 , piece_id=ZC54VVALPOY3IH4OQPUU6DPYFABV63SV33KV45ODP3WUDAOHTIYA, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:50.953386419 (493h4m50.824469454s) [noise=true , address=77.90.31.11:4555 , piece_id=MLANYW6PPT4RXIIV6CWZJEQNC6DHFSENPPNPWOYYCYF7LEBXL2IA, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:51.262930147 (493h4m51.134013182s) [noise=true , address=145.79.227.50:4373 , piece_id=EUSFDA3KK3QRC727PJ2ZBU3OXX7FBJEZ2O4EC2NP52LUXBLGYT5A, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:51.298969812 (493h4m51.170052847s) [noise=true , address=145.79.227.47:4266 , piece_id=O4PKILRO3FBSQUVTU3FLL57FYLZOMIYXEEDZMOAZ7UFM7IEJ3T7Q, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:51.423957922 (493h4m51.295040957s) [noise=true , address=145.79.227.37:4567 , piece_id=HVMMW6PU2JR7P53RCYRG26WGHX4LDYETAKU5FP3FKYGEFDHQBGGA, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:51.654638800 (493h4m51.525721845s) [noise=true , address=145.79.227.19:4079 , piece_id=ZEOFO5JGYYT76IGCHSQY47OR4TFSBAS7SC33QDS44H7RJXKAZZ5Q, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:51.661717759 (493h4m51.532800804s) [noise=true , address=145.79.227.35:4482 , piece_id=EME54IOT2RMJM3CGSXGY4VRBUTDIBSD4PHGYDYDG7BGSQUHKENRA, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:52.470190148 (493h4m52.341273193s) [noise=true , address=145.79.227.49:4338 , piece_id=MMJ5CQ7O6NQGBN7PJ4DOPJCPDXYZVDPATNUY4244GBQSYNLEOOGQ, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:52.876094342 (493h4m52.747177387s) [noise=true , address=145.79.227.36:4517 , piece_id=EP4MV6YHY6NMPDSNCMOEIFYKEJEZEMOGNZ45MP33EYN3QHOXJTEQ, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:52.879955905 (493h4m52.75103897s) [noise=true , address=145.79.227.15:4876 , piece_id=MGPE2NJVQUOSTFVVBTOSYXNV6ZF67YRKWF2TS4KQBV4G5Y6EUC3Q, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:52.879980362 (493h4m52.751063407s) [seg_pos={0 0}, upload=432766]: Segment reached optimal threshold of 13 pieces. Dec 26 04:33:52.880327262 (493h4m52.751410318s) [noise=true , address=145.79.227.44:4195 , piece_id=4IWWRAPTBN7MSSTAABZ4R64MTOMXXOSFL4TQQDTK2RPV6LQIHH5A, seg_pos={0 0}, upload=432766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 04:33:52.880337291 (493h4m52.751420326s) [noise=true , address=145.79.227.44:4195 , piece_id=4IWWRAPTBN7MSSTAABZ4R64MTOMXXOSFL4TQQDTK2RPV6LQIHH5A, seg_pos={0 0}, upload=432766]: Long tail cancellation due to optimal threshold Dec 26 04:33:52.880384371 (493h4m52.751467426s) [noise=true , address=145.79.227.6:4257 , piece_id=DFQDMCW5ONXE3W6QHSQCNQD6SJQPGC7Q643HVE6WBEIPQG64JJFQ, seg_pos={0 0}, upload=432766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: read tcp 188.166.225.170:60046->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 04:33:52.880394200 (493h4m52.751477235s) [noise=true , address=145.79.227.6:4257 , piece_id=DFQDMCW5ONXE3W6QHSQCNQD6SJQPGC7Q643HVE6WBEIPQG64JJFQ, seg_pos={0 0}, upload=432766]: Long tail cancellation due to optimal threshold Dec 26 04:33:52.880513467 (493h4m52.751596512s) [noise=true , address=145.79.227.46:4602 , piece_id=E362APFCNUVQRYRNIKABYFY2NRZZ7WY6GGM5PBBORR3OCUQQ3MPQ, seg_pos={0 0}, upload=432766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d): protocol: read tcp 188.166.225.170:42962->145.79.227.46:4602: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 04:33:52.880520901 (493h4m52.751603946s) [noise=true , address=145.79.227.46:4602 , piece_id=E362APFCNUVQRYRNIKABYFY2NRZZ7WY6GGM5PBBORR3OCUQQ3MPQ, seg_pos={0 0}, upload=432766]: Long tail cancellation due to optimal threshold Dec 26 04:33:52.880540438 (493h4m52.751623483s) [seg_pos={0 0}, upload=432766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:33:52.880626472 (493h4m52.751709517s) [upload=432766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:33:52.880639808 (493h4m52.751722853s) [upload=432766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:33:53.161101307 (493h4m53.032184352s) [upload=432766]: Done uploading segments... Dec 26 04:33:53.161120343 (493h4m53.032203378s) [upload=432766]: Upload finished. err: <nil> Duration: 15.985747469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................X..X..X.X....XX] duration: 30.00341972s trace: https://jaeger.dev.storj.io/trace/21ffd64f6429782 FAIL (2025-12-26 04:34:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: EME54IOT2RMJM3CGSXGY4VRBUTDIBSD4PHGYDYDG7BGSQUHKENRA) context deadline exceeded; piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: HVMMW6PU2JR7P53RCYRG26WGHX4LDYETAKU5FP3FKYGEFDHQBGGA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: MLANYW6PPT4RXIIV6CWZJEQNC6DHFSENPPNPWOYYCYF7LEBXL2IA) context deadline exceeded ======LOGS====== Dec 26 04:33:47.666005705 (493h4m47.53708874s) [upload=432766]: Starting upload Dec 26 04:33:47.666076099 (493h4m47.537159164s) [upload=432766]: Done starting upload Dec 26 04:33:47.666297852 (493h4m47.537380897s) [upload=432766]: Uploading segments... Dec 26 04:33:47.666409384 (493h4m47.537492429s) [upload=432766]: Got next segment. Inline: false Dec 26 04:33:47.666434823 (493h4m47.537517868s) [upload=432766]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:33:47.666441485 (493h4m47.53752452s) [upload=432766]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:33:47.907599031 (493h4m47.778682106s) [seg_pos={0 0}, upload=432766]: Begin upload segment... Dec 26 04:33:47.907652533 (493h4m47.778735568s) [seg_pos={0 0}, upload=432766]: Done begin upload segment. Dec 26 04:33:47.907682350 (493h4m47.778765385s) [noise=true , address=77.90.31.13:4627 , piece_id=AGEDJ42SFMJGNQR6IHWYJ7564VJVPTDVJ4BVKLBS3SYBEGG3262Q, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.907696777 (493h4m47.778779822s) [noise=true , address=77.90.31.11:4555 , piece_id=MLANYW6PPT4RXIIV6CWZJEQNC6DHFSENPPNPWOYYCYF7LEBXL2IA, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.907745601 (493h4m47.778828636s) [noise=true , address=145.79.227.36:4517 , piece_id=EP4MV6YHY6NMPDSNCMOEIFYKEJEZEMOGNZ45MP33EYN3QHOXJTEQ, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.908233830 (493h4m47.779316876s) [noise=true , address=145.79.227.15:4876 , piece_id=MGPE2NJVQUOSTFVVBTOSYXNV6ZF67YRKWF2TS4KQBV4G5Y6EUC3Q, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.908874089 (493h4m47.779957134s) [noise=true , address=77.90.31.10:4519 , piece_id=ZC54VVALPOY3IH4OQPUU6DPYFABV63SV33KV45ODP3WUDAOHTIYA, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.908882936 (493h4m47.779965981s) [noise=true , address=77.90.31.19:4844 , piece_id=AZONQAM7M7ARQFIBFQKUDWU464G3PSNYMHXFDOBAJU36YAHWDVFA, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.909180563 (493h4m47.780263598s) [noise=true , address=145.79.227.35:4482 , piece_id=EME54IOT2RMJM3CGSXGY4VRBUTDIBSD4PHGYDYDG7BGSQUHKENRA, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.909441690 (493h4m47.780524735s) [noise=true , address=145.79.227.49:4338 , piece_id=MMJ5CQ7O6NQGBN7PJ4DOPJCPDXYZVDPATNUY4244GBQSYNLEOOGQ, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.909517794 (493h4m47.78060084s) [noise=true , address=145.79.227.47:4266 , piece_id=O4PKILRO3FBSQUVTU3FLL57FYLZOMIYXEEDZMOAZ7UFM7IEJ3T7Q, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.909893009 (493h4m47.780976054s) [noise=true , address=145.79.227.50:4373 , piece_id=EUSFDA3KK3QRC727PJ2ZBU3OXX7FBJEZ2O4EC2NP52LUXBLGYT5A, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.910006015 (493h4m47.78108906s) [noise=true , address=145.79.227.46:4602 , piece_id=E362APFCNUVQRYRNIKABYFY2NRZZ7WY6GGM5PBBORR3OCUQQ3MPQ, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.910182020 (493h4m47.781265065s) [noise=true , address=145.79.227.44:4195 , piece_id=4IWWRAPTBN7MSSTAABZ4R64MTOMXXOSFL4TQQDTK2RPV6LQIHH5A, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.910668937 (493h4m47.781751982s) [noise=true , address=145.79.227.6:4257 , piece_id=DFQDMCW5ONXE3W6QHSQCNQD6SJQPGC7Q643HVE6WBEIPQG64JJFQ, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.910684847 (493h4m47.781767892s) [noise=true , address=45.91.116.11:4119 , piece_id=XP7753K4QWBLQTE56O4OUL5R7TRKGM2HL667ENUUWKZVOOKK4DWQ, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.911193266 (493h4m47.782276311s) [noise=true , address=145.79.227.37:4567 , piece_id=HVMMW6PU2JR7P53RCYRG26WGHX4LDYETAKU5FP3FKYGEFDHQBGGA, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:47.913028299 (493h4m47.784111344s) [noise=true , address=145.79.227.19:4079 , piece_id=ZEOFO5JGYYT76IGCHSQY47OR4TFSBAS7SC33QDS44H7RJXKAZZ5Q, seg_pos={0 0}, upload=432766]: Uploading piece... Dec 26 04:33:49.771832237 (493h4m49.642915282s) [upload=432766]: Next returned nil segment Dec 26 04:33:49.771848548 (493h4m49.642931613s) [upload=432766]: Waiting for error group managing segments... Dec 26 04:33:50.163660729 (493h4m50.034743775s) [noise=true , address=77.90.31.19:4844 , piece_id=AZONQAM7M7ARQFIBFQKUDWU464G3PSNYMHXFDOBAJU36YAHWDVFA, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:50.244521202 (493h4m50.115604247s) [noise=true , address=45.91.116.11:4119 , piece_id=XP7753K4QWBLQTE56O4OUL5R7TRKGM2HL667ENUUWKZVOOKK4DWQ, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:50.307098661 (493h4m50.178181716s) [noise=true , address=77.90.31.13:4627 , piece_id=AGEDJ42SFMJGNQR6IHWYJ7564VJVPTDVJ4BVKLBS3SYBEGG3262Q, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:50.416730701 (493h4m50.287813736s) [noise=true , address=77.90.31.10:4519 , piece_id=ZC54VVALPOY3IH4OQPUU6DPYFABV63SV33KV45ODP3WUDAOHTIYA, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:50.953386419 (493h4m50.824469454s) [noise=true , address=77.90.31.11:4555 , piece_id=MLANYW6PPT4RXIIV6CWZJEQNC6DHFSENPPNPWOYYCYF7LEBXL2IA, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:51.262930147 (493h4m51.134013182s) [noise=true , address=145.79.227.50:4373 , piece_id=EUSFDA3KK3QRC727PJ2ZBU3OXX7FBJEZ2O4EC2NP52LUXBLGYT5A, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:51.298969812 (493h4m51.170052847s) [noise=true , address=145.79.227.47:4266 , piece_id=O4PKILRO3FBSQUVTU3FLL57FYLZOMIYXEEDZMOAZ7UFM7IEJ3T7Q, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:51.423957922 (493h4m51.295040957s) [noise=true , address=145.79.227.37:4567 , piece_id=HVMMW6PU2JR7P53RCYRG26WGHX4LDYETAKU5FP3FKYGEFDHQBGGA, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:51.654638800 (493h4m51.525721845s) [noise=true , address=145.79.227.19:4079 , piece_id=ZEOFO5JGYYT76IGCHSQY47OR4TFSBAS7SC33QDS44H7RJXKAZZ5Q, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:51.661717759 (493h4m51.532800804s) [noise=true , address=145.79.227.35:4482 , piece_id=EME54IOT2RMJM3CGSXGY4VRBUTDIBSD4PHGYDYDG7BGSQUHKENRA, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:52.470190148 (493h4m52.341273193s) [noise=true , address=145.79.227.49:4338 , piece_id=MMJ5CQ7O6NQGBN7PJ4DOPJCPDXYZVDPATNUY4244GBQSYNLEOOGQ, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:52.876094342 (493h4m52.747177387s) [noise=true , address=145.79.227.36:4517 , piece_id=EP4MV6YHY6NMPDSNCMOEIFYKEJEZEMOGNZ45MP33EYN3QHOXJTEQ, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:52.879955905 (493h4m52.75103897s) [noise=true , address=145.79.227.15:4876 , piece_id=MGPE2NJVQUOSTFVVBTOSYXNV6ZF67YRKWF2TS4KQBV4G5Y6EUC3Q, seg_pos={0 0}, upload=432766]: Done uploading piece. err: <nil> Dec 26 04:33:52.879980362 (493h4m52.751063407s) [seg_pos={0 0}, upload=432766]: Segment reached optimal threshold of 13 pieces. Dec 26 04:33:52.880327262 (493h4m52.751410318s) [noise=true , address=145.79.227.44:4195 , piece_id=4IWWRAPTBN7MSSTAABZ4R64MTOMXXOSFL4TQQDTK2RPV6LQIHH5A, seg_pos={0 0}, upload=432766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 26 04:33:52.880337291 (493h4m52.751420326s) [noise=true , address=145.79.227.44:4195 , piece_id=4IWWRAPTBN7MSSTAABZ4R64MTOMXXOSFL4TQQDTK2RPV6LQIHH5A, seg_pos={0 0}, upload=432766]: Long tail cancellation due to optimal threshold Dec 26 04:33:52.880384371 (493h4m52.751467426s) [noise=true , address=145.79.227.6:4257 , piece_id=DFQDMCW5ONXE3W6QHSQCNQD6SJQPGC7Q643HVE6WBEIPQG64JJFQ, seg_pos={0 0}, upload=432766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: read tcp 188.166.225.170:60046->145.79.227.6:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 04:33:52.880394200 (493h4m52.751477235s) [noise=true , address=145.79.227.6:4257 , piece_id=DFQDMCW5ONXE3W6QHSQCNQD6SJQPGC7Q643HVE6WBEIPQG64JJFQ, seg_pos={0 0}, upload=432766]: Long tail cancellation due to optimal threshold Dec 26 04:33:52.880513467 (493h4m52.751596512s) [noise=true , address=145.79.227.46:4602 , piece_id=E362APFCNUVQRYRNIKABYFY2NRZZ7WY6GGM5PBBORR3OCUQQ3MPQ, seg_pos={0 0}, upload=432766]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d): protocol: read tcp 188.166.225.170:42962->145.79.227.46:4602: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 04:33:52.880520901 (493h4m52.751603946s) [noise=true , address=145.79.227.46:4602 , piece_id=E362APFCNUVQRYRNIKABYFY2NRZZ7WY6GGM5PBBORR3OCUQQ3MPQ, seg_pos={0 0}, upload=432766]: Long tail cancellation due to optimal threshold Dec 26 04:33:52.880540438 (493h4m52.751623483s) [seg_pos={0 0}, upload=432766]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:33:52.880626472 (493h4m52.751709517s) [upload=432766]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:33:52.880639808 (493h4m52.751722853s) [upload=432766]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:33:53.161101307 (493h4m53.032184352s) [upload=432766]: Done uploading segments... Dec 26 04:33:53.161120343 (493h4m53.032203378s) [upload=432766]: Upload finished. err: <nil> Duration: 30.003433357s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.13191991s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:34:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 7.131950078s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................................X..X..X.X....X] duration: 30.240424945s trace: https://jaeger.dev.storj.io/trace/121a951f44f04387 FAIL (2025-12-26 04:33:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 26 04:33:17.216862351 (493h4m17.087945396s) [upload=432759]: Starting upload Dec 26 04:33:17.216933717 (493h4m17.088016762s) [upload=432759]: Done starting upload Dec 26 04:33:17.217063875 (493h4m17.08814692s) [upload=432759]: Uploading segments... Dec 26 04:33:17.217173474 (493h4m17.088256519s) [upload=432759]: Got next segment. Inline: false Dec 26 04:33:17.217248698 (493h4m17.088331743s) [upload=432759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:33:17.217256332 (493h4m17.088339377s) [upload=432759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:33:17.463345390 (493h4m17.334428435s) [seg_pos={0 0}, upload=432759]: Begin upload segment... Dec 26 04:33:17.463395134 (493h4m17.334478179s) [seg_pos={0 0}, upload=432759]: Done begin upload segment. Dec 26 04:33:17.463424380 (493h4m17.334507445s) [noise=true , address=45.91.116.13:4154 , piece_id=CWVNXWLFD52FVKOLT6C2PHIBQFQFKDY6HTEKKLG25VC7XZAHHYQA, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.463434229 (493h4m17.334517264s) [noise=true , address=145.79.227.5:4223 , piece_id=SOYUW4SCIHX4O4V5VJYMBSVLVPIMMM57OGUDE2QLFW7EQLIYV3UA, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.463561641 (493h4m17.334644686s) [noise=true , address=145.79.227.39:4014 , piece_id=DZADGH7KHZZ3LSMAIXJLRUNQULHGV7OYZ5GHQNYCJDYSNMPA3M7Q, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.464494788 (493h4m17.335577853s) [noise=true , address=145.79.227.14:4837 , piece_id=ZKYXSF4QVMI5VIMD27O4DGBWKSRVVTD3FRZUZIXSXFZE5LFR7MOQ, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.465058121 (493h4m17.336141166s) [noise=true , address=145.79.227.17:4972 , piece_id=USUUJLAKU4D5ZRZTROZXQFSIRVJ35BF2LVOUUVA5USX5M5D2T75Q, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.465650308 (493h4m17.336733354s) [noise=true , address=145.79.227.47:4265 , piece_id=PLMDP4CXET53RU7CJHY4WJUWYUIKFW57O6PCK43ZOFUWXNCO3FZQ, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.466599005 (493h4m17.33768205s) [noise=true , address=45.91.116.11:4119 , piece_id=SALVMWOPG4P4A7YFHBOVUCR5K7FYXFISKPSHFBCT6XBGGVCBSHPQ, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.467483059 (493h4m17.338566114s) [noise=true , address=145.79.227.41:4086 , piece_id=3T4T36P2MRFMXRAK6OBANNGHTP4OY527A7YW3FE6YSJKSHNPSUQA, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.469334183 (493h4m17.340417228s) [noise=true , address=145.79.227.13:4803 , piece_id=KVQAWOKHYNY2EXLD4X32HMUAG6HRHB4RXT7WUWIV55ZNN3BBZUYA, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.469633884 (493h4m17.340716929s) [noise=true , address=145.79.227.8:4329 , piece_id=I4Q6MEAZAYOBNMDIC34MITZXEBAY7NMMFFY2XK7DFTTIRERSKM2A, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.471733913 (493h4m17.342816958s) [noise=true , address=145.79.227.27:4689 , piece_id=KAHY5BXMD3QIGWPEYPDK3A7FJWXMZXUM4VY2CIADBX6425HPLWZA, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.473865101 (493h4m17.344948146s) [noise=true , address=145.79.227.49:4337 , piece_id=KDEXATBMIGEJ65PH45NCBAPYTBHWHBIWVYR2MJMVV4FNAAHL67GA, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.476301380 (493h4m17.347384415s) [noise=true , address=145.79.227.15:4876 , piece_id=YSCGQTLRXMSPBSEIBK7H3COAHQEOGVYMVOX25OHTZLBF7FO3KQ7Q, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.478251533 (493h4m17.349334578s) [noise=true , address=45.91.116.9:4263 , piece_id=BIXPO4U6JRIS4EF33KJFMS4ELZXQJT2WTALCNOZSVEBJEP77FG7Q, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.480217027 (493h4m17.351300072s) [noise=true , address=77.90.31.17:4773 , piece_id=NF77PP5S4MC7APQYAI43GGJBHM7F76DEX45OKVQ3PDB5IPFIZBIA, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:17.482993873 (493h4m17.354076918s) [noise=true , address=145.79.227.42:4121 , piece_id=5WRAPMTGB674OB5FUSKFCSXA24FOXKVQZQFN7ZMATUVR56GDR2KQ, seg_pos={0 0}, upload=432759]: Uploading piece... Dec 26 04:33:19.385439653 (493h4m19.256522698s) [upload=432759]: Next returned nil segment Dec 26 04:33:19.385499858 (493h4m19.256582893s) [upload=432759]: Waiting for error group managing segments... Dec 26 04:33:19.689594298 (493h4m19.560677343s) [noise=true , address=77.90.31.17:4773 , piece_id=NF77PP5S4MC7APQYAI43GGJBHM7F76DEX45OKVQ3PDB5IPFIZBIA, seg_pos={0 0}, upload=432759]: Done uploading piece. err: <nil> Dec 26 04:33:19.702001805 (493h4m19.57308484s) [noise=true , address=45.91.116.11:4119 , piece_id=SALVMWOPG4P4A7YFHBOVUCR5K7FYXFISKPSHFBCT6XBGGVCBSHPQ, seg_pos={0 0}, upload=432759]: Done uploading piece. err: <nil> Dec 26 04:33:19.756391993 (493h4m19.627475038s) [noise=true , address=45.91.116.9:4263 , piece_id=BIXPO4U6JRIS4EF33KJFMS4ELZXQJT2WTALCNOZSVEBJEP77FG7Q, seg_pos={0 0}, upload=432759]: Done uploading piece. err: <nil> Dec 26 04:33:20.157379135 (493h4m20.02846217s) [noise=true , address=45.91.116.13:4154 , piece_id=CWVNXWLFD52FVKOLT6C2PHIBQFQFKDY6HTEKKLG25VC7XZAHHYQA, seg_pos={0 0}, upload=432759]: Done uploading piece. err: <nil> Dec 26 04:33:20.415215723 (493h4m20.286298758s) [noise=true , address=145.79.227.13:4803 , piece_id=KVQAWOKHYNY2EXLD4X32HMUAG6HRHB4RXT7WUWIV55ZNN3BBZUYA, seg_pos={0 0}, upload=432759]: Done uploading piece. err: <nil> Dec 26 04:33:20.621398601 (493h4m20.492481647s) [noise=true , address=145.79.227.15:4876 , piece_id=YSCGQTLRXMSPBSEIBK7H3COAHQEOGVYMVOX25OHTZLBF7FO3KQ7Q, seg_pos={0 0}, upload=432759]: Done uploading piece. err: <nil> Dec 26 04:33:20.796954471 (493h4m20.668037506s) [noise=true , address=145.79.227.49:4337 , piece_id=KDEXATBMIGEJ65PH45NCBAPYTBHWHBIWVYR2MJMVV4FNAAHL67GA, seg_pos={0 0}, upload=432759]: Done uploading piece. err: <nil> Dec 26 04:33:20.993473961 (493h4m20.864556996s) [noise=true , address=145.79.227.47:4265 , piece_id=PLMDP4CXET53RU7CJHY4WJUWYUIKFW57O6PCK43ZOFUWXNCO3FZQ, seg_pos={0 0}, upload=432759]: Done uploading piece. err: <nil> Dec 26 04:33:21.099478282 (493h4m20.970561327s) [noise=true , address=145.79.227.39:4014 , piece_id=DZADGH7KHZZ3LSMAIXJLRUNQULHGV7OYZ5GHQNYCJDYSNMPA3M7Q, seg_pos={0 0}, upload=432759]: Done uploading piece. err: <nil> Dec 26 04:33:21.152319040 (493h4m21.023402075s) [noise=true , address=145.79.227.5:4223 , piece_id=SOYUW4SCIHX4O4V5VJYMBSVLVPIMMM57OGUDE2QLFW7EQLIYV3UA, seg_pos={0 0}, upload=432759]: Done uploading piece. err: <nil> Dec 26 04:33:21.178519561 (493h4m21.049602596s) [noise=true , address=145.79.227.41:4086 , piece_id=3T4T36P2MRFMXRAK6OBANNGHTP4OY527A7YW3FE6YSJKSHNPSUQA, seg_pos={0 0}, upload=432759]: Done uploading piece. err: <nil> Dec 26 04:33:21.857434136 (493h4m21.728517181s) [noise=true , address=145.79.227.27:4689 , piece_id=KAHY5BXMD3QIGWPEYPDK3A7FJWXMZXUM4VY2CIADBX6425HPLWZA, seg_pos={0 0}, upload=432759]: Done uploading piece. err: <nil> Dec 26 04:33:46.531779393 (493h4m46.402862438s) [noise=true , address=145.79.227.14:4837 , piece_id=ZKYXSF4QVMI5VIMD27O4DGBWKSRVVTD3FRZUZIXSXFZE5LFR7MOQ, seg_pos={0 0}, upload=432759]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 188.166.225.170:58778->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 04:33:46.531926062 (493h4m46.403009107s) [noise=true , address=145.79.227.17:4972 , piece_id=USUUJLAKU4D5ZRZTROZXQFSIRVJ35BF2LVOUUVA5USX5M5D2T75Q, seg_pos={0 0}, upload=432759]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: read tcp 188.166.225.170:39390->145.79.227.17:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 04:33:46.531998470 (493h4m46.403081505s) [noise=true , address=145.79.227.8:4329 , piece_id=I4Q6MEAZAYOBNMDIC34MITZXEBAY7NMMFFY2XK7DFTTIRERSKM2A, seg_pos={0 0}, upload=432759]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:60480->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 04:33:46.532036393 (493h4m46.403119428s) [noise=true , address=145.79.227.42:4121 , piece_id=5WRAPMTGB674OB5FUSKFCSXA24FOXKVQZQFN7ZMATUVR56GDR2KQ, seg_pos={0 0}, upload=432759]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:37668->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 04:33:46.532062752 (493h4m46.403145787s) [seg_pos={0 0}, upload=432759]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 26 04:33:46.770408766 (493h4m46.641491811s) [upload=432759]: Done uploading segments... Dec 26 04:33:46.770448371 (493h4m46.641531417s) [upload=432759]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.240442549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 33.438412283s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:33:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 33.438441487s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.401283116s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:33:09 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.401320328s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.040281343s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:32:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.040303395s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 17.671113474s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:32:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 17.671133472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................X..X..X.X....] duration: 6.087892063s trace: https://jaeger.dev.storj.io/trace/e90c181730b031b FAIL (2025-12-26 04:32:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ) context deadline exceeded ======LOGS====== Dec 26 04:29:00.554092578 (493h0m0.425175623s) [upload=432698]: Starting upload Dec 26 04:29:00.554144296 (493h0m0.425227331s) [upload=432698]: Done starting upload Dec 26 04:29:00.554358104 (493h0m0.425441139s) [upload=432698]: Uploading segments... Dec 26 04:29:00.554397098 (493h0m0.425480153s) [upload=432698]: Got next segment. Inline: false Dec 26 04:29:00.554426384 (493h0m0.425509419s) [upload=432698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:29:00.554430852 (493h0m0.425513897s) [upload=432698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:29:00.798652143 (493h0m0.669735178s) [seg_pos={0 0}, upload=432698]: Begin upload segment... Dec 26 04:29:00.798701186 (493h0m0.669784231s) [seg_pos={0 0}, upload=432698]: Done begin upload segment. Dec 26 04:29:00.798727296 (493h0m0.669810341s) [noise=true , address=145.79.227.4:4187 , piece_id=CWF3DUWJQH7COKXAWPT54V2DF5KUDPTADDWGLFFME4CFSOHYVITA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.799932019 (493h0m0.671015054s) [noise=true , address=145.79.227.11:4730 , piece_id=NT7NENR3SZGN4YU6PODYQLAO3BHYUQW7EJ35XBVD7D23S73LOYIQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.801306277 (493h0m0.672389312s) [noise=true , address=145.79.227.16:4910 , piece_id=B3U6DUSA2JYU7TWTHGFTHNO7BWOZOBQPSF3G4C2U3JCPH4NHIZPQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.803663556 (493h0m0.674746591s) [noise=true , address=145.79.227.45:4231 , piece_id=PSCQ5Q5HIR5XVNNWZCUWC4FO3VXPTWHO3NMD2GU6OU2BAH6R3HUQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.803923089 (493h0m0.675006124s) [noise=true , address=145.79.227.50:4374 , piece_id=NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.806454510 (493h0m0.677537545s) [noise=true , address=45.91.116.12:4228 , piece_id=JOKIMO3J5LQWLGHMFKLBY2AMI5LMGVRZIDO7VJRQRQK6K6KYQYZQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.806665962 (493h0m0.677748997s) [noise=true , address=45.91.116.10:4084 , piece_id=5CXF3C2DBSJ2JZFFXLCLONUUAGFLZ7ZLGHW46TRVFVTAO2UQ43WQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.807851029 (493h0m0.678934064s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.809043611 (493h0m0.680126646s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.809693357 (493h0m0.680776402s) [noise=true , address=145.79.227.2:4113 , piece_id=UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.810617597 (493h0m0.681700633s) [noise=true , address=145.79.227.35:4483 , piece_id=BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.811208843 (493h0m0.682291878s) [noise=true , address=45.91.116.8:4047 , piece_id=B6JIPX6CDKGPRCDGMGBPW6SX36OCUBEA7YCU7QNDORN2MJXTWNHA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.814516812 (493h0m0.685599847s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.816312091 (493h0m0.687395156s) [noise=true , address=145.79.227.36:4518 , piece_id=YGXVNUXGQ5RVZJLOQMWQSH242PQ6UHFPPATWKKXHSIOYHAICYDCQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.816668730 (493h0m0.687751765s) [noise=true , address=145.79.227.6:4257 , piece_id=VUVJ2OU3JWAANIIRE2SMN2VHXKNRMI5HQBSGAWARR3PPX4WNLK7Q, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.817653655 (493h0m0.68873669s) [noise=true , address=145.79.227.32:4835 , piece_id=PPWUY6JGBUZBBJFYUH4IPID2SD2NZEQDKNGNPEA5YTIRMNLIGTQQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:02.660580911 (493h0m2.531663966s) [upload=432698]: Next returned nil segment Dec 26 04:29:02.660597361 (493h0m2.531680406s) [upload=432698]: Waiting for error group managing segments... Dec 26 04:29:02.979231108 (493h0m2.850314153s) [noise=true , address=145.79.227.16:4910 , piece_id=B3U6DUSA2JYU7TWTHGFTHNO7BWOZOBQPSF3G4C2U3JCPH4NHIZPQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.980055678 (493h0m2.851138713s) [noise=true , address=145.79.227.35:4483 , piece_id=BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.986612804 (493h0m2.857695859s) [noise=true , address=145.79.227.50:4374 , piece_id=NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.991019165 (493h0m2.86210221s) [noise=true , address=145.79.227.11:4730 , piece_id=NT7NENR3SZGN4YU6PODYQLAO3BHYUQW7EJ35XBVD7D23S73LOYIQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.991949846 (493h0m2.863032891s) [noise=true , address=145.79.227.36:4518 , piece_id=YGXVNUXGQ5RVZJLOQMWQSH242PQ6UHFPPATWKKXHSIOYHAICYDCQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.995666403 (493h0m2.866749448s) [noise=true , address=145.79.227.2:4113 , piece_id=UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.005181933 (493h0m2.876264988s) [noise=true , address=145.79.227.45:4231 , piece_id=PSCQ5Q5HIR5XVNNWZCUWC4FO3VXPTWHO3NMD2GU6OU2BAH6R3HUQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.008829569 (493h0m2.879912644s) [noise=true , address=145.79.227.6:4257 , piece_id=VUVJ2OU3JWAANIIRE2SMN2VHXKNRMI5HQBSGAWARR3PPX4WNLK7Q, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.174186057 (493h0m3.045269102s) [noise=true , address=45.91.116.12:4228 , piece_id=JOKIMO3J5LQWLGHMFKLBY2AMI5LMGVRZIDO7VJRQRQK6K6KYQYZQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.180159612 (493h0m3.051242647s) [noise=true , address=45.91.116.8:4047 , piece_id=B6JIPX6CDKGPRCDGMGBPW6SX36OCUBEA7YCU7QNDORN2MJXTWNHA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.275183554 (493h0m3.1462666s) [noise=true , address=145.79.227.32:4835 , piece_id=PPWUY6JGBUZBBJFYUH4IPID2SD2NZEQDKNGNPEA5YTIRMNLIGTQQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.365360287 (493h0m3.236443332s) [noise=true , address=45.91.116.10:4084 , piece_id=5CXF3C2DBSJ2JZFFXLCLONUUAGFLZ7ZLGHW46TRVFVTAO2UQ43WQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.661406362 (493h0m3.532489408s) [noise=true , address=145.79.227.4:4187 , piece_id=CWF3DUWJQH7COKXAWPT54V2DF5KUDPTADDWGLFFME4CFSOHYVITA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.661434306 (493h0m3.532517341s) [seg_pos={0 0}, upload=432698]: Segment reached optimal threshold of 13 pieces. Dec 26 04:29:03.661676347 (493h0m3.532759402s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 04:29:03.661696134 (493h0m3.532779179s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662148616 (493h0m3.533231661s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 04:29:03.662159156 (493h0m3.533242191s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662279194 (493h0m3.53336227s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection Dec 26 04:29:03.662370429 (493h0m3.533453504s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662416115 (493h0m3.53349916s) [seg_pos={0 0}, upload=432698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:29:03.662580849 (493h0m3.533663904s) [upload=432698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:29:03.662589375 (493h0m3.53367242s) [upload=432698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:29:03.952779680 (493h0m3.823862725s) [upload=432698]: Done uploading segments... Dec 26 04:29:03.952803276 (493h0m3.823886321s) [upload=432698]: Upload finished. err: <nil> Duration: 6.087919054s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 14.308627757s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:31:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 14.308655439s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 34.151536629s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:31:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 34.151563911s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 11.399936342s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:31:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 11.400013479s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................X..X..X.X...] duration: 6.062182116s trace: https://jaeger.dev.storj.io/trace/e90c181730b031b FAIL (2025-12-26 04:31:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ) context deadline exceeded ======LOGS====== Dec 26 04:29:00.554092578 (493h0m0.425175623s) [upload=432698]: Starting upload Dec 26 04:29:00.554144296 (493h0m0.425227331s) [upload=432698]: Done starting upload Dec 26 04:29:00.554358104 (493h0m0.425441139s) [upload=432698]: Uploading segments... Dec 26 04:29:00.554397098 (493h0m0.425480153s) [upload=432698]: Got next segment. Inline: false Dec 26 04:29:00.554426384 (493h0m0.425509419s) [upload=432698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:29:00.554430852 (493h0m0.425513897s) [upload=432698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:29:00.798652143 (493h0m0.669735178s) [seg_pos={0 0}, upload=432698]: Begin upload segment... Dec 26 04:29:00.798701186 (493h0m0.669784231s) [seg_pos={0 0}, upload=432698]: Done begin upload segment. Dec 26 04:29:00.798727296 (493h0m0.669810341s) [noise=true , address=145.79.227.4:4187 , piece_id=CWF3DUWJQH7COKXAWPT54V2DF5KUDPTADDWGLFFME4CFSOHYVITA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.799932019 (493h0m0.671015054s) [noise=true , address=145.79.227.11:4730 , piece_id=NT7NENR3SZGN4YU6PODYQLAO3BHYUQW7EJ35XBVD7D23S73LOYIQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.801306277 (493h0m0.672389312s) [noise=true , address=145.79.227.16:4910 , piece_id=B3U6DUSA2JYU7TWTHGFTHNO7BWOZOBQPSF3G4C2U3JCPH4NHIZPQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.803663556 (493h0m0.674746591s) [noise=true , address=145.79.227.45:4231 , piece_id=PSCQ5Q5HIR5XVNNWZCUWC4FO3VXPTWHO3NMD2GU6OU2BAH6R3HUQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.803923089 (493h0m0.675006124s) [noise=true , address=145.79.227.50:4374 , piece_id=NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.806454510 (493h0m0.677537545s) [noise=true , address=45.91.116.12:4228 , piece_id=JOKIMO3J5LQWLGHMFKLBY2AMI5LMGVRZIDO7VJRQRQK6K6KYQYZQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.806665962 (493h0m0.677748997s) [noise=true , address=45.91.116.10:4084 , piece_id=5CXF3C2DBSJ2JZFFXLCLONUUAGFLZ7ZLGHW46TRVFVTAO2UQ43WQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.807851029 (493h0m0.678934064s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.809043611 (493h0m0.680126646s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.809693357 (493h0m0.680776402s) [noise=true , address=145.79.227.2:4113 , piece_id=UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.810617597 (493h0m0.681700633s) [noise=true , address=145.79.227.35:4483 , piece_id=BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.811208843 (493h0m0.682291878s) [noise=true , address=45.91.116.8:4047 , piece_id=B6JIPX6CDKGPRCDGMGBPW6SX36OCUBEA7YCU7QNDORN2MJXTWNHA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.814516812 (493h0m0.685599847s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.816312091 (493h0m0.687395156s) [noise=true , address=145.79.227.36:4518 , piece_id=YGXVNUXGQ5RVZJLOQMWQSH242PQ6UHFPPATWKKXHSIOYHAICYDCQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.816668730 (493h0m0.687751765s) [noise=true , address=145.79.227.6:4257 , piece_id=VUVJ2OU3JWAANIIRE2SMN2VHXKNRMI5HQBSGAWARR3PPX4WNLK7Q, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.817653655 (493h0m0.68873669s) [noise=true , address=145.79.227.32:4835 , piece_id=PPWUY6JGBUZBBJFYUH4IPID2SD2NZEQDKNGNPEA5YTIRMNLIGTQQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:02.660580911 (493h0m2.531663966s) [upload=432698]: Next returned nil segment Dec 26 04:29:02.660597361 (493h0m2.531680406s) [upload=432698]: Waiting for error group managing segments... Dec 26 04:29:02.979231108 (493h0m2.850314153s) [noise=true , address=145.79.227.16:4910 , piece_id=B3U6DUSA2JYU7TWTHGFTHNO7BWOZOBQPSF3G4C2U3JCPH4NHIZPQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.980055678 (493h0m2.851138713s) [noise=true , address=145.79.227.35:4483 , piece_id=BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.986612804 (493h0m2.857695859s) [noise=true , address=145.79.227.50:4374 , piece_id=NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.991019165 (493h0m2.86210221s) [noise=true , address=145.79.227.11:4730 , piece_id=NT7NENR3SZGN4YU6PODYQLAO3BHYUQW7EJ35XBVD7D23S73LOYIQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.991949846 (493h0m2.863032891s) [noise=true , address=145.79.227.36:4518 , piece_id=YGXVNUXGQ5RVZJLOQMWQSH242PQ6UHFPPATWKKXHSIOYHAICYDCQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.995666403 (493h0m2.866749448s) [noise=true , address=145.79.227.2:4113 , piece_id=UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.005181933 (493h0m2.876264988s) [noise=true , address=145.79.227.45:4231 , piece_id=PSCQ5Q5HIR5XVNNWZCUWC4FO3VXPTWHO3NMD2GU6OU2BAH6R3HUQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.008829569 (493h0m2.879912644s) [noise=true , address=145.79.227.6:4257 , piece_id=VUVJ2OU3JWAANIIRE2SMN2VHXKNRMI5HQBSGAWARR3PPX4WNLK7Q, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.174186057 (493h0m3.045269102s) [noise=true , address=45.91.116.12:4228 , piece_id=JOKIMO3J5LQWLGHMFKLBY2AMI5LMGVRZIDO7VJRQRQK6K6KYQYZQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.180159612 (493h0m3.051242647s) [noise=true , address=45.91.116.8:4047 , piece_id=B6JIPX6CDKGPRCDGMGBPW6SX36OCUBEA7YCU7QNDORN2MJXTWNHA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.275183554 (493h0m3.1462666s) [noise=true , address=145.79.227.32:4835 , piece_id=PPWUY6JGBUZBBJFYUH4IPID2SD2NZEQDKNGNPEA5YTIRMNLIGTQQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.365360287 (493h0m3.236443332s) [noise=true , address=45.91.116.10:4084 , piece_id=5CXF3C2DBSJ2JZFFXLCLONUUAGFLZ7ZLGHW46TRVFVTAO2UQ43WQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.661406362 (493h0m3.532489408s) [noise=true , address=145.79.227.4:4187 , piece_id=CWF3DUWJQH7COKXAWPT54V2DF5KUDPTADDWGLFFME4CFSOHYVITA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.661434306 (493h0m3.532517341s) [seg_pos={0 0}, upload=432698]: Segment reached optimal threshold of 13 pieces. Dec 26 04:29:03.661676347 (493h0m3.532759402s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 04:29:03.661696134 (493h0m3.532779179s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662148616 (493h0m3.533231661s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 04:29:03.662159156 (493h0m3.533242191s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662279194 (493h0m3.53336227s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection Dec 26 04:29:03.662370429 (493h0m3.533453504s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662416115 (493h0m3.53349916s) [seg_pos={0 0}, upload=432698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:29:03.662580849 (493h0m3.533663904s) [upload=432698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:29:03.662589375 (493h0m3.53367242s) [upload=432698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:29:03.952779680 (493h0m3.823862725s) [upload=432698]: Done uploading segments... Dec 26 04:29:03.952803276 (493h0m3.823886321s) [upload=432698]: Upload finished. err: <nil> Duration: 6.062227673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 35.126741304s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:30:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 35.126774346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................X..X..X.X..] duration: 6.016254885s trace: https://jaeger.dev.storj.io/trace/e90c181730b031b FAIL (2025-12-26 04:30:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ) context deadline exceeded ======LOGS====== Dec 26 04:29:00.554092578 (493h0m0.425175623s) [upload=432698]: Starting upload Dec 26 04:29:00.554144296 (493h0m0.425227331s) [upload=432698]: Done starting upload Dec 26 04:29:00.554358104 (493h0m0.425441139s) [upload=432698]: Uploading segments... Dec 26 04:29:00.554397098 (493h0m0.425480153s) [upload=432698]: Got next segment. Inline: false Dec 26 04:29:00.554426384 (493h0m0.425509419s) [upload=432698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:29:00.554430852 (493h0m0.425513897s) [upload=432698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:29:00.798652143 (493h0m0.669735178s) [seg_pos={0 0}, upload=432698]: Begin upload segment... Dec 26 04:29:00.798701186 (493h0m0.669784231s) [seg_pos={0 0}, upload=432698]: Done begin upload segment. Dec 26 04:29:00.798727296 (493h0m0.669810341s) [noise=true , address=145.79.227.4:4187 , piece_id=CWF3DUWJQH7COKXAWPT54V2DF5KUDPTADDWGLFFME4CFSOHYVITA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.799932019 (493h0m0.671015054s) [noise=true , address=145.79.227.11:4730 , piece_id=NT7NENR3SZGN4YU6PODYQLAO3BHYUQW7EJ35XBVD7D23S73LOYIQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.801306277 (493h0m0.672389312s) [noise=true , address=145.79.227.16:4910 , piece_id=B3U6DUSA2JYU7TWTHGFTHNO7BWOZOBQPSF3G4C2U3JCPH4NHIZPQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.803663556 (493h0m0.674746591s) [noise=true , address=145.79.227.45:4231 , piece_id=PSCQ5Q5HIR5XVNNWZCUWC4FO3VXPTWHO3NMD2GU6OU2BAH6R3HUQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.803923089 (493h0m0.675006124s) [noise=true , address=145.79.227.50:4374 , piece_id=NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.806454510 (493h0m0.677537545s) [noise=true , address=45.91.116.12:4228 , piece_id=JOKIMO3J5LQWLGHMFKLBY2AMI5LMGVRZIDO7VJRQRQK6K6KYQYZQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.806665962 (493h0m0.677748997s) [noise=true , address=45.91.116.10:4084 , piece_id=5CXF3C2DBSJ2JZFFXLCLONUUAGFLZ7ZLGHW46TRVFVTAO2UQ43WQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.807851029 (493h0m0.678934064s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.809043611 (493h0m0.680126646s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.809693357 (493h0m0.680776402s) [noise=true , address=145.79.227.2:4113 , piece_id=UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.810617597 (493h0m0.681700633s) [noise=true , address=145.79.227.35:4483 , piece_id=BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.811208843 (493h0m0.682291878s) [noise=true , address=45.91.116.8:4047 , piece_id=B6JIPX6CDKGPRCDGMGBPW6SX36OCUBEA7YCU7QNDORN2MJXTWNHA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.814516812 (493h0m0.685599847s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.816312091 (493h0m0.687395156s) [noise=true , address=145.79.227.36:4518 , piece_id=YGXVNUXGQ5RVZJLOQMWQSH242PQ6UHFPPATWKKXHSIOYHAICYDCQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.816668730 (493h0m0.687751765s) [noise=true , address=145.79.227.6:4257 , piece_id=VUVJ2OU3JWAANIIRE2SMN2VHXKNRMI5HQBSGAWARR3PPX4WNLK7Q, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.817653655 (493h0m0.68873669s) [noise=true , address=145.79.227.32:4835 , piece_id=PPWUY6JGBUZBBJFYUH4IPID2SD2NZEQDKNGNPEA5YTIRMNLIGTQQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:02.660580911 (493h0m2.531663966s) [upload=432698]: Next returned nil segment Dec 26 04:29:02.660597361 (493h0m2.531680406s) [upload=432698]: Waiting for error group managing segments... Dec 26 04:29:02.979231108 (493h0m2.850314153s) [noise=true , address=145.79.227.16:4910 , piece_id=B3U6DUSA2JYU7TWTHGFTHNO7BWOZOBQPSF3G4C2U3JCPH4NHIZPQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.980055678 (493h0m2.851138713s) [noise=true , address=145.79.227.35:4483 , piece_id=BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.986612804 (493h0m2.857695859s) [noise=true , address=145.79.227.50:4374 , piece_id=NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.991019165 (493h0m2.86210221s) [noise=true , address=145.79.227.11:4730 , piece_id=NT7NENR3SZGN4YU6PODYQLAO3BHYUQW7EJ35XBVD7D23S73LOYIQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.991949846 (493h0m2.863032891s) [noise=true , address=145.79.227.36:4518 , piece_id=YGXVNUXGQ5RVZJLOQMWQSH242PQ6UHFPPATWKKXHSIOYHAICYDCQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.995666403 (493h0m2.866749448s) [noise=true , address=145.79.227.2:4113 , piece_id=UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.005181933 (493h0m2.876264988s) [noise=true , address=145.79.227.45:4231 , piece_id=PSCQ5Q5HIR5XVNNWZCUWC4FO3VXPTWHO3NMD2GU6OU2BAH6R3HUQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.008829569 (493h0m2.879912644s) [noise=true , address=145.79.227.6:4257 , piece_id=VUVJ2OU3JWAANIIRE2SMN2VHXKNRMI5HQBSGAWARR3PPX4WNLK7Q, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.174186057 (493h0m3.045269102s) [noise=true , address=45.91.116.12:4228 , piece_id=JOKIMO3J5LQWLGHMFKLBY2AMI5LMGVRZIDO7VJRQRQK6K6KYQYZQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.180159612 (493h0m3.051242647s) [noise=true , address=45.91.116.8:4047 , piece_id=B6JIPX6CDKGPRCDGMGBPW6SX36OCUBEA7YCU7QNDORN2MJXTWNHA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.275183554 (493h0m3.1462666s) [noise=true , address=145.79.227.32:4835 , piece_id=PPWUY6JGBUZBBJFYUH4IPID2SD2NZEQDKNGNPEA5YTIRMNLIGTQQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.365360287 (493h0m3.236443332s) [noise=true , address=45.91.116.10:4084 , piece_id=5CXF3C2DBSJ2JZFFXLCLONUUAGFLZ7ZLGHW46TRVFVTAO2UQ43WQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.661406362 (493h0m3.532489408s) [noise=true , address=145.79.227.4:4187 , piece_id=CWF3DUWJQH7COKXAWPT54V2DF5KUDPTADDWGLFFME4CFSOHYVITA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.661434306 (493h0m3.532517341s) [seg_pos={0 0}, upload=432698]: Segment reached optimal threshold of 13 pieces. Dec 26 04:29:03.661676347 (493h0m3.532759402s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 04:29:03.661696134 (493h0m3.532779179s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662148616 (493h0m3.533231661s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 04:29:03.662159156 (493h0m3.533242191s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662279194 (493h0m3.53336227s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection Dec 26 04:29:03.662370429 (493h0m3.533453504s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662416115 (493h0m3.53349916s) [seg_pos={0 0}, upload=432698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:29:03.662580849 (493h0m3.533663904s) [upload=432698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:29:03.662589375 (493h0m3.53367242s) [upload=432698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:29:03.952779680 (493h0m3.823862725s) [upload=432698]: Done uploading segments... Dec 26 04:29:03.952803276 (493h0m3.823886321s) [upload=432698]: Upload finished. err: <nil> Duration: 6.0162734s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 7.939957527s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:30:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 7.939988146s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................X..X..X.X.] duration: 21.460160332s trace: https://jaeger.dev.storj.io/trace/e90c181730b031b FAIL (2025-12-26 04:29:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ) context deadline exceeded ======LOGS====== Dec 26 04:29:00.554092578 (493h0m0.425175623s) [upload=432698]: Starting upload Dec 26 04:29:00.554144296 (493h0m0.425227331s) [upload=432698]: Done starting upload Dec 26 04:29:00.554358104 (493h0m0.425441139s) [upload=432698]: Uploading segments... Dec 26 04:29:00.554397098 (493h0m0.425480153s) [upload=432698]: Got next segment. Inline: false Dec 26 04:29:00.554426384 (493h0m0.425509419s) [upload=432698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:29:00.554430852 (493h0m0.425513897s) [upload=432698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:29:00.798652143 (493h0m0.669735178s) [seg_pos={0 0}, upload=432698]: Begin upload segment... Dec 26 04:29:00.798701186 (493h0m0.669784231s) [seg_pos={0 0}, upload=432698]: Done begin upload segment. Dec 26 04:29:00.798727296 (493h0m0.669810341s) [noise=true , address=145.79.227.4:4187 , piece_id=CWF3DUWJQH7COKXAWPT54V2DF5KUDPTADDWGLFFME4CFSOHYVITA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.799932019 (493h0m0.671015054s) [noise=true , address=145.79.227.11:4730 , piece_id=NT7NENR3SZGN4YU6PODYQLAO3BHYUQW7EJ35XBVD7D23S73LOYIQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.801306277 (493h0m0.672389312s) [noise=true , address=145.79.227.16:4910 , piece_id=B3U6DUSA2JYU7TWTHGFTHNO7BWOZOBQPSF3G4C2U3JCPH4NHIZPQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.803663556 (493h0m0.674746591s) [noise=true , address=145.79.227.45:4231 , piece_id=PSCQ5Q5HIR5XVNNWZCUWC4FO3VXPTWHO3NMD2GU6OU2BAH6R3HUQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.803923089 (493h0m0.675006124s) [noise=true , address=145.79.227.50:4374 , piece_id=NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.806454510 (493h0m0.677537545s) [noise=true , address=45.91.116.12:4228 , piece_id=JOKIMO3J5LQWLGHMFKLBY2AMI5LMGVRZIDO7VJRQRQK6K6KYQYZQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.806665962 (493h0m0.677748997s) [noise=true , address=45.91.116.10:4084 , piece_id=5CXF3C2DBSJ2JZFFXLCLONUUAGFLZ7ZLGHW46TRVFVTAO2UQ43WQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.807851029 (493h0m0.678934064s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.809043611 (493h0m0.680126646s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.809693357 (493h0m0.680776402s) [noise=true , address=145.79.227.2:4113 , piece_id=UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.810617597 (493h0m0.681700633s) [noise=true , address=145.79.227.35:4483 , piece_id=BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.811208843 (493h0m0.682291878s) [noise=true , address=45.91.116.8:4047 , piece_id=B6JIPX6CDKGPRCDGMGBPW6SX36OCUBEA7YCU7QNDORN2MJXTWNHA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.814516812 (493h0m0.685599847s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.816312091 (493h0m0.687395156s) [noise=true , address=145.79.227.36:4518 , piece_id=YGXVNUXGQ5RVZJLOQMWQSH242PQ6UHFPPATWKKXHSIOYHAICYDCQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.816668730 (493h0m0.687751765s) [noise=true , address=145.79.227.6:4257 , piece_id=VUVJ2OU3JWAANIIRE2SMN2VHXKNRMI5HQBSGAWARR3PPX4WNLK7Q, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.817653655 (493h0m0.68873669s) [noise=true , address=145.79.227.32:4835 , piece_id=PPWUY6JGBUZBBJFYUH4IPID2SD2NZEQDKNGNPEA5YTIRMNLIGTQQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:02.660580911 (493h0m2.531663966s) [upload=432698]: Next returned nil segment Dec 26 04:29:02.660597361 (493h0m2.531680406s) [upload=432698]: Waiting for error group managing segments... Dec 26 04:29:02.979231108 (493h0m2.850314153s) [noise=true , address=145.79.227.16:4910 , piece_id=B3U6DUSA2JYU7TWTHGFTHNO7BWOZOBQPSF3G4C2U3JCPH4NHIZPQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.980055678 (493h0m2.851138713s) [noise=true , address=145.79.227.35:4483 , piece_id=BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.986612804 (493h0m2.857695859s) [noise=true , address=145.79.227.50:4374 , piece_id=NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.991019165 (493h0m2.86210221s) [noise=true , address=145.79.227.11:4730 , piece_id=NT7NENR3SZGN4YU6PODYQLAO3BHYUQW7EJ35XBVD7D23S73LOYIQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.991949846 (493h0m2.863032891s) [noise=true , address=145.79.227.36:4518 , piece_id=YGXVNUXGQ5RVZJLOQMWQSH242PQ6UHFPPATWKKXHSIOYHAICYDCQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.995666403 (493h0m2.866749448s) [noise=true , address=145.79.227.2:4113 , piece_id=UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.005181933 (493h0m2.876264988s) [noise=true , address=145.79.227.45:4231 , piece_id=PSCQ5Q5HIR5XVNNWZCUWC4FO3VXPTWHO3NMD2GU6OU2BAH6R3HUQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.008829569 (493h0m2.879912644s) [noise=true , address=145.79.227.6:4257 , piece_id=VUVJ2OU3JWAANIIRE2SMN2VHXKNRMI5HQBSGAWARR3PPX4WNLK7Q, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.174186057 (493h0m3.045269102s) [noise=true , address=45.91.116.12:4228 , piece_id=JOKIMO3J5LQWLGHMFKLBY2AMI5LMGVRZIDO7VJRQRQK6K6KYQYZQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.180159612 (493h0m3.051242647s) [noise=true , address=45.91.116.8:4047 , piece_id=B6JIPX6CDKGPRCDGMGBPW6SX36OCUBEA7YCU7QNDORN2MJXTWNHA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.275183554 (493h0m3.1462666s) [noise=true , address=145.79.227.32:4835 , piece_id=PPWUY6JGBUZBBJFYUH4IPID2SD2NZEQDKNGNPEA5YTIRMNLIGTQQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.365360287 (493h0m3.236443332s) [noise=true , address=45.91.116.10:4084 , piece_id=5CXF3C2DBSJ2JZFFXLCLONUUAGFLZ7ZLGHW46TRVFVTAO2UQ43WQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.661406362 (493h0m3.532489408s) [noise=true , address=145.79.227.4:4187 , piece_id=CWF3DUWJQH7COKXAWPT54V2DF5KUDPTADDWGLFFME4CFSOHYVITA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.661434306 (493h0m3.532517341s) [seg_pos={0 0}, upload=432698]: Segment reached optimal threshold of 13 pieces. Dec 26 04:29:03.661676347 (493h0m3.532759402s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 04:29:03.661696134 (493h0m3.532779179s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662148616 (493h0m3.533231661s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 04:29:03.662159156 (493h0m3.533242191s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662279194 (493h0m3.53336227s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection Dec 26 04:29:03.662370429 (493h0m3.533453504s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662416115 (493h0m3.53349916s) [seg_pos={0 0}, upload=432698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:29:03.662580849 (493h0m3.533663904s) [upload=432698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:29:03.662589375 (493h0m3.53367242s) [upload=432698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:29:03.952779680 (493h0m3.823862725s) [upload=432698]: Done uploading segments... Dec 26 04:29:03.952803276 (493h0m3.823886321s) [upload=432698]: Upload finished. err: <nil> Duration: 21.460192404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 29.76115407s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:29:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 29.761193294s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.065968226s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:29:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.065997041s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................X..X..X.X] duration: 30.003040864s trace: https://jaeger.dev.storj.io/trace/e90c181730b031b FAIL (2025-12-26 04:29:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, Piece ID: NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ) context deadline exceeded ======LOGS====== Dec 26 04:29:00.554092578 (493h0m0.425175623s) [upload=432698]: Starting upload Dec 26 04:29:00.554144296 (493h0m0.425227331s) [upload=432698]: Done starting upload Dec 26 04:29:00.554358104 (493h0m0.425441139s) [upload=432698]: Uploading segments... Dec 26 04:29:00.554397098 (493h0m0.425480153s) [upload=432698]: Got next segment. Inline: false Dec 26 04:29:00.554426384 (493h0m0.425509419s) [upload=432698]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:29:00.554430852 (493h0m0.425513897s) [upload=432698]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:29:00.798652143 (493h0m0.669735178s) [seg_pos={0 0}, upload=432698]: Begin upload segment... Dec 26 04:29:00.798701186 (493h0m0.669784231s) [seg_pos={0 0}, upload=432698]: Done begin upload segment. Dec 26 04:29:00.798727296 (493h0m0.669810341s) [noise=true , address=145.79.227.4:4187 , piece_id=CWF3DUWJQH7COKXAWPT54V2DF5KUDPTADDWGLFFME4CFSOHYVITA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.799932019 (493h0m0.671015054s) [noise=true , address=145.79.227.11:4730 , piece_id=NT7NENR3SZGN4YU6PODYQLAO3BHYUQW7EJ35XBVD7D23S73LOYIQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.801306277 (493h0m0.672389312s) [noise=true , address=145.79.227.16:4910 , piece_id=B3U6DUSA2JYU7TWTHGFTHNO7BWOZOBQPSF3G4C2U3JCPH4NHIZPQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.803663556 (493h0m0.674746591s) [noise=true , address=145.79.227.45:4231 , piece_id=PSCQ5Q5HIR5XVNNWZCUWC4FO3VXPTWHO3NMD2GU6OU2BAH6R3HUQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.803923089 (493h0m0.675006124s) [noise=true , address=145.79.227.50:4374 , piece_id=NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.806454510 (493h0m0.677537545s) [noise=true , address=45.91.116.12:4228 , piece_id=JOKIMO3J5LQWLGHMFKLBY2AMI5LMGVRZIDO7VJRQRQK6K6KYQYZQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.806665962 (493h0m0.677748997s) [noise=true , address=45.91.116.10:4084 , piece_id=5CXF3C2DBSJ2JZFFXLCLONUUAGFLZ7ZLGHW46TRVFVTAO2UQ43WQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.807851029 (493h0m0.678934064s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.809043611 (493h0m0.680126646s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.809693357 (493h0m0.680776402s) [noise=true , address=145.79.227.2:4113 , piece_id=UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.810617597 (493h0m0.681700633s) [noise=true , address=145.79.227.35:4483 , piece_id=BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.811208843 (493h0m0.682291878s) [noise=true , address=45.91.116.8:4047 , piece_id=B6JIPX6CDKGPRCDGMGBPW6SX36OCUBEA7YCU7QNDORN2MJXTWNHA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.814516812 (493h0m0.685599847s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.816312091 (493h0m0.687395156s) [noise=true , address=145.79.227.36:4518 , piece_id=YGXVNUXGQ5RVZJLOQMWQSH242PQ6UHFPPATWKKXHSIOYHAICYDCQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.816668730 (493h0m0.687751765s) [noise=true , address=145.79.227.6:4257 , piece_id=VUVJ2OU3JWAANIIRE2SMN2VHXKNRMI5HQBSGAWARR3PPX4WNLK7Q, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:00.817653655 (493h0m0.68873669s) [noise=true , address=145.79.227.32:4835 , piece_id=PPWUY6JGBUZBBJFYUH4IPID2SD2NZEQDKNGNPEA5YTIRMNLIGTQQ, seg_pos={0 0}, upload=432698]: Uploading piece... Dec 26 04:29:02.660580911 (493h0m2.531663966s) [upload=432698]: Next returned nil segment Dec 26 04:29:02.660597361 (493h0m2.531680406s) [upload=432698]: Waiting for error group managing segments... Dec 26 04:29:02.979231108 (493h0m2.850314153s) [noise=true , address=145.79.227.16:4910 , piece_id=B3U6DUSA2JYU7TWTHGFTHNO7BWOZOBQPSF3G4C2U3JCPH4NHIZPQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.980055678 (493h0m2.851138713s) [noise=true , address=145.79.227.35:4483 , piece_id=BZXVPMB2EJGVANJE5A27UXG7ITFBMF2CRXCOJMKE52UL4FQIJFPQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.986612804 (493h0m2.857695859s) [noise=true , address=145.79.227.50:4374 , piece_id=NTLW7AVT6SSBT4VA7B6QHUJF23TTF4JTAXXXXGDDMVW6RQ5N5GNQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.991019165 (493h0m2.86210221s) [noise=true , address=145.79.227.11:4730 , piece_id=NT7NENR3SZGN4YU6PODYQLAO3BHYUQW7EJ35XBVD7D23S73LOYIQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.991949846 (493h0m2.863032891s) [noise=true , address=145.79.227.36:4518 , piece_id=YGXVNUXGQ5RVZJLOQMWQSH242PQ6UHFPPATWKKXHSIOYHAICYDCQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:02.995666403 (493h0m2.866749448s) [noise=true , address=145.79.227.2:4113 , piece_id=UUV6TAWZR27AG6S4K572ZBMKHFPGFJ2LATSK3JWLK5Z3K33LOUAA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.005181933 (493h0m2.876264988s) [noise=true , address=145.79.227.45:4231 , piece_id=PSCQ5Q5HIR5XVNNWZCUWC4FO3VXPTWHO3NMD2GU6OU2BAH6R3HUQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.008829569 (493h0m2.879912644s) [noise=true , address=145.79.227.6:4257 , piece_id=VUVJ2OU3JWAANIIRE2SMN2VHXKNRMI5HQBSGAWARR3PPX4WNLK7Q, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.174186057 (493h0m3.045269102s) [noise=true , address=45.91.116.12:4228 , piece_id=JOKIMO3J5LQWLGHMFKLBY2AMI5LMGVRZIDO7VJRQRQK6K6KYQYZQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.180159612 (493h0m3.051242647s) [noise=true , address=45.91.116.8:4047 , piece_id=B6JIPX6CDKGPRCDGMGBPW6SX36OCUBEA7YCU7QNDORN2MJXTWNHA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.275183554 (493h0m3.1462666s) [noise=true , address=145.79.227.32:4835 , piece_id=PPWUY6JGBUZBBJFYUH4IPID2SD2NZEQDKNGNPEA5YTIRMNLIGTQQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.365360287 (493h0m3.236443332s) [noise=true , address=45.91.116.10:4084 , piece_id=5CXF3C2DBSJ2JZFFXLCLONUUAGFLZ7ZLGHW46TRVFVTAO2UQ43WQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.661406362 (493h0m3.532489408s) [noise=true , address=145.79.227.4:4187 , piece_id=CWF3DUWJQH7COKXAWPT54V2DF5KUDPTADDWGLFFME4CFSOHYVITA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: <nil> Dec 26 04:29:03.661434306 (493h0m3.532517341s) [seg_pos={0 0}, upload=432698]: Segment reached optimal threshold of 13 pieces. Dec 26 04:29:03.661676347 (493h0m3.532759402s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 04:29:03.661696134 (493h0m3.532779179s) [noise=true , address=45.91.116.9:4264 , piece_id=KOELG57WULULFLKN6DUXFOM4UV2P5JDJSYBJTP4QW5XJBRA6ZJLA, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662148616 (493h0m3.533231661s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 26 04:29:03.662159156 (493h0m3.533242191s) [noise=true , address=45.91.116.14:4191 , piece_id=CPUONXMUH3FHCE3UYAZI75P2FC2B6MHFKOX6FOB4ISZJMXCXGKEQ, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662279194 (493h0m3.53336227s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43482->145.79.227.39:4013: use of closed network connection Dec 26 04:29:03.662370429 (493h0m3.533453504s) [noise=true , address=145.79.227.39:4013 , piece_id=MMNTAG7I3G7UC2K7FHKO5ZYKSQVL4XU3TE4WS5XDVCICJGWPE37Q, seg_pos={0 0}, upload=432698]: Long tail cancellation due to optimal threshold Dec 26 04:29:03.662416115 (493h0m3.53349916s) [seg_pos={0 0}, upload=432698]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:29:03.662580849 (493h0m3.533663904s) [upload=432698]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:29:03.662589375 (493h0m3.53367242s) [upload=432698]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:29:03.952779680 (493h0m3.823862725s) [upload=432698]: Done uploading segments... Dec 26 04:29:03.952803276 (493h0m3.823886321s) [upload=432698]: Upload finished. err: <nil> Duration: 30.003054139s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)