Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 22.688046065s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 05:00:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 22.688076621s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [........X..X..X.X....XX.X.X.X.XX.X......X.....X.....XXX.....] duration: 6.237312259s trace: https://jaeger.dev.storj.io/trace/35f5df21c5c50271 FAIL (2025-12-26 05:00:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28m0.364691594s) [upload=433115]: Uploading segments... Dec 26 04:57:00.493817967 (493h28m0.364901002s) [upload=433115]: Got next segment. Inline: false Dec 26 04:57:00.493858234 (493h28m0.364941269s) [upload=433115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:57:00.493863213 (493h28m0.364946248s) [upload=433115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:57:00.740808295 (493h28m0.61189134s) [seg_pos={0 0}, upload=433115]: Begin upload segment... Dec 26 04:57:00.740867859 (493h28m0.611950904s) [seg_pos={0 0}, upload=433115]: Done begin upload segment. Dec 26 04:57:00.740903106 (493h28m0.611986151s) [noise=true , address=77.90.31.10:4519 , piece_id=CU7ZQ7UCOAEHIA4K5DWXKRIQLUTCJHTZWT6EP3FGEL62P4HXC5KA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.740956066 (493h28m0.612039111s) [noise=true , address=145.79.227.36:4518 , piece_id=YDL62RACOFXZBL5MNHRAM5RXYB3MGIOD5OZRNIS5B2BJUCWBH7RA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.741498009 (493h28m0.612581054s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.741964597 (493h28m0.613047642s) [noise=true , address=45.91.116.6:4974 , piece_id=B2E335J5GMUAMWF3NUGYMYSFEIKMIQ6HWIQI5COWLZ3XEBF4RI7A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.742495277 (493h28m0.613578322s) [noise=true , address=145.79.227.38:4978 , piece_id=REDW7YRN2GTHZP3X2XNBXYAYSZZ7RCAYGMFTKDF2NXRZRWBJCXZQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.742922181 (493h28m0.614005226s) [noise=true , address=145.79.227.17:4970 , piece_id=LHZJF36OFDUX3LARKFPSL5AAHHGIZ4NXNX7RE2XLC4YSOM6UKSVQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.743365033 (493h28m0.614448078s) [noise=true , address=45.91.116.10:4082 , piece_id=FT57Q64IUDACKAYHGFG5CTYCVP6E2FQ7SFMLZFJUJKYL5SIR5UPA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.743851219 (493h28m0.614934264s) [noise=true , address=145.79.227.35:4482 , piece_id=BAPUKYK7CIULIZW2TVAUD4GWLE66X5S3AEMCYTK2XUGC3YSNW5LA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.745195890 (493h28m0.616278925s) [noise=true , address=145.79.227.15:4873 , piece_id=DWJPRCDVDMTNBBD7IUVMXXMOWAIEKAQSLDC7QZ56SCSUUSXUT36A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.747342197 (493h28m0.618425242s) [noise=true , address=145.79.227.29:4941 , piece_id=4D4F6QUU27M4SHX5EBOOZ23ZOCGJFEGD53PQYKNR5EHAOBWXDC7Q, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.747535324 (493h28m0.618618369s) [noise=true , address=145.79.227.46:4601 , piece_id=X64PRZMCWUULW76CGTW6KIFC45PCXZKDD6IWCR55K267OIMTFYVQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.750206982 (493h28m0.621290017s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.752203423 (493h28m0.623286458s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.754550804 (493h28m0.625633839s) [noise=true , address=145.79.227.41:4087 , piece_id=7FDVP5YAUBC3EQUW7BVJL4YWBBRCJJRJFHXBNPITQUSHSAUZS7BA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.755805773 (493h28m0.626888808s) [noise=true , address=145.79.227.42:4122 , piece_id=LZUFOCM2IQFREBOE6O5ZDBXSEMIDACH7B7YDH37TFEHT7SKEDL6Q, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.759546145 (493h28m0.63062919s) [noise=true , address=145.79.227.19:4079 , piece_id=WE423SOQQPRWFL4XCLW3V3MT5FRPJN2XQDBOYJNQ5PSMND3DNDCA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:02.378510547 (493h28m2.249593582s) [upload=433115]: Next returned nil segment Dec 26 04:57:02.378544913 (493h28m2.249627958s) [upload=433115]: Waiting for error group managing segments... Dec 26 04:57:02.758822895 (493h28m2.62990593s) [noise=true , address=145.79.227.15:4873 , piece_id=DWJPRCDVDMTNBBD7IUVMXXMOWAIEKAQSLDC7QZ56SCSUUSXUT36A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.758943015 (493h28m2.63002606s) [noise=true , address=145.79.227.41:4087 , piece_id=7FDVP5YAUBC3EQUW7BVJL4YWBBRCJJRJFHXBNPITQUSHSAUZS7BA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.763261908 (493h28m2.634344943s) [noise=true , address=145.79.227.29:4941 , piece_id=4D4F6QUU27M4SHX5EBOOZ23ZOCGJFEGD53PQYKNR5EHAOBWXDC7Q, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.764096297 (493h28m2.635179332s) [noise=true , address=145.79.227.17:4970 , piece_id=LHZJF36OFDUX3LARKFPSL5AAHHGIZ4NXNX7RE2XLC4YSOM6UKSVQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.766834852 (493h28m2.637917887s) [noise=true , address=145.79.227.42:4122 , piece_id=LZUFOCM2IQFREBOE6O5ZDBXSEMIDACH7B7YDH37TFEHT7SKEDL6Q, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.771434841 (493h28m2.642517876s) [noise=true , address=145.79.227.36:4518 , piece_id=YDL62RACOFXZBL5MNHRAM5RXYB3MGIOD5OZRNIS5B2BJUCWBH7RA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.771917460 (493h28m2.643000505s) [noise=true , address=145.79.227.19:4079 , piece_id=WE423SOQQPRWFL4XCLW3V3MT5FRPJN2XQDBOYJNQ5PSMND3DNDCA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.778998393 (493h28m2.650081428s) [noise=true , address=145.79.227.38:4978 , piece_id=REDW7YRN2GTHZP3X2XNBXYAYSZZ7RCAYGMFTKDF2NXRZRWBJCXZQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.781024683 (493h28m2.652107718s) [noise=true , address=145.79.227.46:4601 , piece_id=X64PRZMCWUULW76CGTW6KIFC45PCXZKDD6IWCR55K267OIMTFYVQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.782549526 (493h28m2.653632561s) [noise=true , address=145.79.227.35:4482 , piece_id=BAPUKYK7CIULIZW2TVAUD4GWLE66X5S3AEMCYTK2XUGC3YSNW5LA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.009543416 (493h28m2.880626461s) [noise=true , address=45.91.116.10:4082 , piece_id=FT57Q64IUDACKAYHGFG5CTYCVP6E2FQ7SFMLZFJUJKYL5SIR5UPA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.015927503 (493h28m2.887010538s) [noise=true , address=45.91.116.6:4974 , piece_id=B2E335J5GMUAMWF3NUGYMYSFEIKMIQ6HWIQI5COWLZ3XEBF4RI7A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.028581768 (493h28m2.899664813s) [noise=true , address=77.90.31.10:4519 , piece_id=CU7ZQ7UCOAEHIA4K5DWXKRIQLUTCJHTZWT6EP3FGEL62P4HXC5KA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.028609521 (493h28m2.899692557s) [seg_pos={0 0}, upload=433115]: Segment reached optimal threshold of 13 pieces. Dec 26 04:57:03.029032897 (493h28m2.900115943s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection Dec 26 04:57:03.029041354 (493h28m2.900124399s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.029625316 (493h28m2.900708361s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection Dec 26 04:57:03.029637359 (493h28m2.900720404s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.030032972 (493h28m2.901116017s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 04:57:03.030047539 (493h28m2.901130585s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.030073750 (493h28m2.901156785s) [seg_pos={0 0}, upload=433115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:57:03.030140557 (493h28m2.901223592s) [upload=433115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:57:03.030147891 (493h28m2.901230936s) [upload=433115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:57:03.326703811 (493h28m3.197786856s) [upload=433115]: Done uploading segments... Dec 26 04:57:03.326795645 (493h28m3.19787869s) [upload=433115]: Upload finished. err: <nil> Duration: 6.23734405s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 18.942076781s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:59:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 18.942092261s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.053058035s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:59:35 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.053087631s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........X..X..X.X....XX.X.X.X.XX.X......X.....X.....XXX....] duration: 6.140180233s trace: https://jaeger.dev.storj.io/trace/35f5df21c5c50271 FAIL (2025-12-26 04:59:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28m0.364691594s) [upload=433115]: Uploading segments... Dec 26 04:57:00.493817967 (493h28m0.364901002s) [upload=433115]: Got next segment. Inline: false Dec 26 04:57:00.493858234 (493h28m0.364941269s) [upload=433115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:57:00.493863213 (493h28m0.364946248s) [upload=433115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:57:00.740808295 (493h28m0.61189134s) [seg_pos={0 0}, upload=433115]: Begin upload segment... Dec 26 04:57:00.740867859 (493h28m0.611950904s) [seg_pos={0 0}, upload=433115]: Done begin upload segment. Dec 26 04:57:00.740903106 (493h28m0.611986151s) [noise=true , address=77.90.31.10:4519 , piece_id=CU7ZQ7UCOAEHIA4K5DWXKRIQLUTCJHTZWT6EP3FGEL62P4HXC5KA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.740956066 (493h28m0.612039111s) [noise=true , address=145.79.227.36:4518 , piece_id=YDL62RACOFXZBL5MNHRAM5RXYB3MGIOD5OZRNIS5B2BJUCWBH7RA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.741498009 (493h28m0.612581054s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.741964597 (493h28m0.613047642s) [noise=true , address=45.91.116.6:4974 , piece_id=B2E335J5GMUAMWF3NUGYMYSFEIKMIQ6HWIQI5COWLZ3XEBF4RI7A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.742495277 (493h28m0.613578322s) [noise=true , address=145.79.227.38:4978 , piece_id=REDW7YRN2GTHZP3X2XNBXYAYSZZ7RCAYGMFTKDF2NXRZRWBJCXZQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.742922181 (493h28m0.614005226s) [noise=true , address=145.79.227.17:4970 , piece_id=LHZJF36OFDUX3LARKFPSL5AAHHGIZ4NXNX7RE2XLC4YSOM6UKSVQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.743365033 (493h28m0.614448078s) [noise=true , address=45.91.116.10:4082 , piece_id=FT57Q64IUDACKAYHGFG5CTYCVP6E2FQ7SFMLZFJUJKYL5SIR5UPA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.743851219 (493h28m0.614934264s) [noise=true , address=145.79.227.35:4482 , piece_id=BAPUKYK7CIULIZW2TVAUD4GWLE66X5S3AEMCYTK2XUGC3YSNW5LA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.745195890 (493h28m0.616278925s) [noise=true , address=145.79.227.15:4873 , piece_id=DWJPRCDVDMTNBBD7IUVMXXMOWAIEKAQSLDC7QZ56SCSUUSXUT36A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.747342197 (493h28m0.618425242s) [noise=true , address=145.79.227.29:4941 , piece_id=4D4F6QUU27M4SHX5EBOOZ23ZOCGJFEGD53PQYKNR5EHAOBWXDC7Q, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.747535324 (493h28m0.618618369s) [noise=true , address=145.79.227.46:4601 , piece_id=X64PRZMCWUULW76CGTW6KIFC45PCXZKDD6IWCR55K267OIMTFYVQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.750206982 (493h28m0.621290017s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.752203423 (493h28m0.623286458s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.754550804 (493h28m0.625633839s) [noise=true , address=145.79.227.41:4087 , piece_id=7FDVP5YAUBC3EQUW7BVJL4YWBBRCJJRJFHXBNPITQUSHSAUZS7BA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.755805773 (493h28m0.626888808s) [noise=true , address=145.79.227.42:4122 , piece_id=LZUFOCM2IQFREBOE6O5ZDBXSEMIDACH7B7YDH37TFEHT7SKEDL6Q, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.759546145 (493h28m0.63062919s) [noise=true , address=145.79.227.19:4079 , piece_id=WE423SOQQPRWFL4XCLW3V3MT5FRPJN2XQDBOYJNQ5PSMND3DNDCA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:02.378510547 (493h28m2.249593582s) [upload=433115]: Next returned nil segment Dec 26 04:57:02.378544913 (493h28m2.249627958s) [upload=433115]: Waiting for error group managing segments... Dec 26 04:57:02.758822895 (493h28m2.62990593s) [noise=true , address=145.79.227.15:4873 , piece_id=DWJPRCDVDMTNBBD7IUVMXXMOWAIEKAQSLDC7QZ56SCSUUSXUT36A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.758943015 (493h28m2.63002606s) [noise=true , address=145.79.227.41:4087 , piece_id=7FDVP5YAUBC3EQUW7BVJL4YWBBRCJJRJFHXBNPITQUSHSAUZS7BA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.763261908 (493h28m2.634344943s) [noise=true , address=145.79.227.29:4941 , piece_id=4D4F6QUU27M4SHX5EBOOZ23ZOCGJFEGD53PQYKNR5EHAOBWXDC7Q, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.764096297 (493h28m2.635179332s) [noise=true , address=145.79.227.17:4970 , piece_id=LHZJF36OFDUX3LARKFPSL5AAHHGIZ4NXNX7RE2XLC4YSOM6UKSVQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.766834852 (493h28m2.637917887s) [noise=true , address=145.79.227.42:4122 , piece_id=LZUFOCM2IQFREBOE6O5ZDBXSEMIDACH7B7YDH37TFEHT7SKEDL6Q, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.771434841 (493h28m2.642517876s) [noise=true , address=145.79.227.36:4518 , piece_id=YDL62RACOFXZBL5MNHRAM5RXYB3MGIOD5OZRNIS5B2BJUCWBH7RA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.771917460 (493h28m2.643000505s) [noise=true , address=145.79.227.19:4079 , piece_id=WE423SOQQPRWFL4XCLW3V3MT5FRPJN2XQDBOYJNQ5PSMND3DNDCA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.778998393 (493h28m2.650081428s) [noise=true , address=145.79.227.38:4978 , piece_id=REDW7YRN2GTHZP3X2XNBXYAYSZZ7RCAYGMFTKDF2NXRZRWBJCXZQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.781024683 (493h28m2.652107718s) [noise=true , address=145.79.227.46:4601 , piece_id=X64PRZMCWUULW76CGTW6KIFC45PCXZKDD6IWCR55K267OIMTFYVQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.782549526 (493h28m2.653632561s) [noise=true , address=145.79.227.35:4482 , piece_id=BAPUKYK7CIULIZW2TVAUD4GWLE66X5S3AEMCYTK2XUGC3YSNW5LA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.009543416 (493h28m2.880626461s) [noise=true , address=45.91.116.10:4082 , piece_id=FT57Q64IUDACKAYHGFG5CTYCVP6E2FQ7SFMLZFJUJKYL5SIR5UPA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.015927503 (493h28m2.887010538s) [noise=true , address=45.91.116.6:4974 , piece_id=B2E335J5GMUAMWF3NUGYMYSFEIKMIQ6HWIQI5COWLZ3XEBF4RI7A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.028581768 (493h28m2.899664813s) [noise=true , address=77.90.31.10:4519 , piece_id=CU7ZQ7UCOAEHIA4K5DWXKRIQLUTCJHTZWT6EP3FGEL62P4HXC5KA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.028609521 (493h28m2.899692557s) [seg_pos={0 0}, upload=433115]: Segment reached optimal threshold of 13 pieces. Dec 26 04:57:03.029032897 (493h28m2.900115943s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection Dec 26 04:57:03.029041354 (493h28m2.900124399s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.029625316 (493h28m2.900708361s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection Dec 26 04:57:03.029637359 (493h28m2.900720404s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.030032972 (493h28m2.901116017s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 04:57:03.030047539 (493h28m2.901130585s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.030073750 (493h28m2.901156785s) [seg_pos={0 0}, upload=433115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:57:03.030140557 (493h28m2.901223592s) [upload=433115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:57:03.030147891 (493h28m2.901230936s) [upload=433115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:57:03.326703811 (493h28m3.197786856s) [upload=433115]: Done uploading segments... Dec 26 04:57:03.326795645 (493h28m3.19787869s) [upload=433115]: Upload finished. err: <nil> Duration: 6.140207576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 26.514913247s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:59:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 26.514943405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..........X..X..X.X....XX.X.X.X.XX.X......X.....X.....XXX...] duration: 22.747185639s trace: https://jaeger.dev.storj.io/trace/35f5df21c5c50271 FAIL (2025-12-26 04:59:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28m0.364691594s) [upload=433115]: Uploading segments... Dec 26 04:57:00.493817967 (493h28m0.364901002s) [upload=433115]: Got next segment. Inline: false Dec 26 04:57:00.493858234 (493h28m0.364941269s) [upload=433115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:57:00.493863213 (493h28m0.364946248s) [upload=433115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:57:00.740808295 (493h28m0.61189134s) [seg_pos={0 0}, upload=433115]: Begin upload segment... Dec 26 04:57:00.740867859 (493h28m0.611950904s) [seg_pos={0 0}, upload=433115]: Done begin upload segment. Dec 26 04:57:00.740903106 (493h28m0.611986151s) [noise=true , address=77.90.31.10:4519 , piece_id=CU7ZQ7UCOAEHIA4K5DWXKRIQLUTCJHTZWT6EP3FGEL62P4HXC5KA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.740956066 (493h28m0.612039111s) [noise=true , address=145.79.227.36:4518 , piece_id=YDL62RACOFXZBL5MNHRAM5RXYB3MGIOD5OZRNIS5B2BJUCWBH7RA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.741498009 (493h28m0.612581054s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.741964597 (493h28m0.613047642s) [noise=true , address=45.91.116.6:4974 , piece_id=B2E335J5GMUAMWF3NUGYMYSFEIKMIQ6HWIQI5COWLZ3XEBF4RI7A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.742495277 (493h28m0.613578322s) [noise=true , address=145.79.227.38:4978 , piece_id=REDW7YRN2GTHZP3X2XNBXYAYSZZ7RCAYGMFTKDF2NXRZRWBJCXZQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.742922181 (493h28m0.614005226s) [noise=true , address=145.79.227.17:4970 , piece_id=LHZJF36OFDUX3LARKFPSL5AAHHGIZ4NXNX7RE2XLC4YSOM6UKSVQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.743365033 (493h28m0.614448078s) [noise=true , address=45.91.116.10:4082 , piece_id=FT57Q64IUDACKAYHGFG5CTYCVP6E2FQ7SFMLZFJUJKYL5SIR5UPA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.743851219 (493h28m0.614934264s) [noise=true , address=145.79.227.35:4482 , piece_id=BAPUKYK7CIULIZW2TVAUD4GWLE66X5S3AEMCYTK2XUGC3YSNW5LA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.745195890 (493h28m0.616278925s) [noise=true , address=145.79.227.15:4873 , piece_id=DWJPRCDVDMTNBBD7IUVMXXMOWAIEKAQSLDC7QZ56SCSUUSXUT36A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.747342197 (493h28m0.618425242s) [noise=true , address=145.79.227.29:4941 , piece_id=4D4F6QUU27M4SHX5EBOOZ23ZOCGJFEGD53PQYKNR5EHAOBWXDC7Q, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.747535324 (493h28m0.618618369s) [noise=true , address=145.79.227.46:4601 , piece_id=X64PRZMCWUULW76CGTW6KIFC45PCXZKDD6IWCR55K267OIMTFYVQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.750206982 (493h28m0.621290017s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.752203423 (493h28m0.623286458s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.754550804 (493h28m0.625633839s) [noise=true , address=145.79.227.41:4087 , piece_id=7FDVP5YAUBC3EQUW7BVJL4YWBBRCJJRJFHXBNPITQUSHSAUZS7BA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.755805773 (493h28m0.626888808s) [noise=true , address=145.79.227.42:4122 , piece_id=LZUFOCM2IQFREBOE6O5ZDBXSEMIDACH7B7YDH37TFEHT7SKEDL6Q, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.759546145 (493h28m0.63062919s) [noise=true , address=145.79.227.19:4079 , piece_id=WE423SOQQPRWFL4XCLW3V3MT5FRPJN2XQDBOYJNQ5PSMND3DNDCA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:02.378510547 (493h28m2.249593582s) [upload=433115]: Next returned nil segment Dec 26 04:57:02.378544913 (493h28m2.249627958s) [upload=433115]: Waiting for error group managing segments... Dec 26 04:57:02.758822895 (493h28m2.62990593s) [noise=true , address=145.79.227.15:4873 , piece_id=DWJPRCDVDMTNBBD7IUVMXXMOWAIEKAQSLDC7QZ56SCSUUSXUT36A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.758943015 (493h28m2.63002606s) [noise=true , address=145.79.227.41:4087 , piece_id=7FDVP5YAUBC3EQUW7BVJL4YWBBRCJJRJFHXBNPITQUSHSAUZS7BA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.763261908 (493h28m2.634344943s) [noise=true , address=145.79.227.29:4941 , piece_id=4D4F6QUU27M4SHX5EBOOZ23ZOCGJFEGD53PQYKNR5EHAOBWXDC7Q, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.764096297 (493h28m2.635179332s) [noise=true , address=145.79.227.17:4970 , piece_id=LHZJF36OFDUX3LARKFPSL5AAHHGIZ4NXNX7RE2XLC4YSOM6UKSVQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.766834852 (493h28m2.637917887s) [noise=true , address=145.79.227.42:4122 , piece_id=LZUFOCM2IQFREBOE6O5ZDBXSEMIDACH7B7YDH37TFEHT7SKEDL6Q, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.771434841 (493h28m2.642517876s) [noise=true , address=145.79.227.36:4518 , piece_id=YDL62RACOFXZBL5MNHRAM5RXYB3MGIOD5OZRNIS5B2BJUCWBH7RA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.771917460 (493h28m2.643000505s) [noise=true , address=145.79.227.19:4079 , piece_id=WE423SOQQPRWFL4XCLW3V3MT5FRPJN2XQDBOYJNQ5PSMND3DNDCA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.778998393 (493h28m2.650081428s) [noise=true , address=145.79.227.38:4978 , piece_id=REDW7YRN2GTHZP3X2XNBXYAYSZZ7RCAYGMFTKDF2NXRZRWBJCXZQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.781024683 (493h28m2.652107718s) [noise=true , address=145.79.227.46:4601 , piece_id=X64PRZMCWUULW76CGTW6KIFC45PCXZKDD6IWCR55K267OIMTFYVQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.782549526 (493h28m2.653632561s) [noise=true , address=145.79.227.35:4482 , piece_id=BAPUKYK7CIULIZW2TVAUD4GWLE66X5S3AEMCYTK2XUGC3YSNW5LA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.009543416 (493h28m2.880626461s) [noise=true , address=45.91.116.10:4082 , piece_id=FT57Q64IUDACKAYHGFG5CTYCVP6E2FQ7SFMLZFJUJKYL5SIR5UPA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.015927503 (493h28m2.887010538s) [noise=true , address=45.91.116.6:4974 , piece_id=B2E335J5GMUAMWF3NUGYMYSFEIKMIQ6HWIQI5COWLZ3XEBF4RI7A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.028581768 (493h28m2.899664813s) [noise=true , address=77.90.31.10:4519 , piece_id=CU7ZQ7UCOAEHIA4K5DWXKRIQLUTCJHTZWT6EP3FGEL62P4HXC5KA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.028609521 (493h28m2.899692557s) [seg_pos={0 0}, upload=433115]: Segment reached optimal threshold of 13 pieces. Dec 26 04:57:03.029032897 (493h28m2.900115943s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection Dec 26 04:57:03.029041354 (493h28m2.900124399s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.029625316 (493h28m2.900708361s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection Dec 26 04:57:03.029637359 (493h28m2.900720404s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.030032972 (493h28m2.901116017s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 04:57:03.030047539 (493h28m2.901130585s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.030073750 (493h28m2.901156785s) [seg_pos={0 0}, upload=433115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:57:03.030140557 (493h28m2.901223592s) [upload=433115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:57:03.030147891 (493h28m2.901230936s) [upload=433115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:57:03.326703811 (493h28m3.197786856s) [upload=433115]: Done uploading segments... Dec 26 04:57:03.326795645 (493h28m3.19787869s) [upload=433115]: Upload finished. err: <nil> Duration: 22.747210885s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.88946285s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:58:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 5.889488108s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 24.803237506s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:58:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 24.803262696s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........X..X..X.X....XX.X.X.X.XX.X......X.....X.....XXX..] duration: 6.162880417s trace: https://jaeger.dev.storj.io/trace/35f5df21c5c50271 FAIL (2025-12-26 04:58:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28m0.364691594s) [upload=433115]: Uploading segments... Dec 26 04:57:00.493817967 (493h28m0.364901002s) [upload=433115]: Got next segment. Inline: false Dec 26 04:57:00.493858234 (493h28m0.364941269s) [upload=433115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:57:00.493863213 (493h28m0.364946248s) [upload=433115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:57:00.740808295 (493h28m0.61189134s) [seg_pos={0 0}, upload=433115]: Begin upload segment... Dec 26 04:57:00.740867859 (493h28m0.611950904s) [seg_pos={0 0}, upload=433115]: Done begin upload segment. Dec 26 04:57:00.740903106 (493h28m0.611986151s) [noise=true , address=77.90.31.10:4519 , piece_id=CU7ZQ7UCOAEHIA4K5DWXKRIQLUTCJHTZWT6EP3FGEL62P4HXC5KA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.740956066 (493h28m0.612039111s) [noise=true , address=145.79.227.36:4518 , piece_id=YDL62RACOFXZBL5MNHRAM5RXYB3MGIOD5OZRNIS5B2BJUCWBH7RA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.741498009 (493h28m0.612581054s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.741964597 (493h28m0.613047642s) [noise=true , address=45.91.116.6:4974 , piece_id=B2E335J5GMUAMWF3NUGYMYSFEIKMIQ6HWIQI5COWLZ3XEBF4RI7A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.742495277 (493h28m0.613578322s) [noise=true , address=145.79.227.38:4978 , piece_id=REDW7YRN2GTHZP3X2XNBXYAYSZZ7RCAYGMFTKDF2NXRZRWBJCXZQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.742922181 (493h28m0.614005226s) [noise=true , address=145.79.227.17:4970 , piece_id=LHZJF36OFDUX3LARKFPSL5AAHHGIZ4NXNX7RE2XLC4YSOM6UKSVQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.743365033 (493h28m0.614448078s) [noise=true , address=45.91.116.10:4082 , piece_id=FT57Q64IUDACKAYHGFG5CTYCVP6E2FQ7SFMLZFJUJKYL5SIR5UPA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.743851219 (493h28m0.614934264s) [noise=true , address=145.79.227.35:4482 , piece_id=BAPUKYK7CIULIZW2TVAUD4GWLE66X5S3AEMCYTK2XUGC3YSNW5LA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.745195890 (493h28m0.616278925s) [noise=true , address=145.79.227.15:4873 , piece_id=DWJPRCDVDMTNBBD7IUVMXXMOWAIEKAQSLDC7QZ56SCSUUSXUT36A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.747342197 (493h28m0.618425242s) [noise=true , address=145.79.227.29:4941 , piece_id=4D4F6QUU27M4SHX5EBOOZ23ZOCGJFEGD53PQYKNR5EHAOBWXDC7Q, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.747535324 (493h28m0.618618369s) [noise=true , address=145.79.227.46:4601 , piece_id=X64PRZMCWUULW76CGTW6KIFC45PCXZKDD6IWCR55K267OIMTFYVQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.750206982 (493h28m0.621290017s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.752203423 (493h28m0.623286458s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.754550804 (493h28m0.625633839s) [noise=true , address=145.79.227.41:4087 , piece_id=7FDVP5YAUBC3EQUW7BVJL4YWBBRCJJRJFHXBNPITQUSHSAUZS7BA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.755805773 (493h28m0.626888808s) [noise=true , address=145.79.227.42:4122 , piece_id=LZUFOCM2IQFREBOE6O5ZDBXSEMIDACH7B7YDH37TFEHT7SKEDL6Q, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.759546145 (493h28m0.63062919s) [noise=true , address=145.79.227.19:4079 , piece_id=WE423SOQQPRWFL4XCLW3V3MT5FRPJN2XQDBOYJNQ5PSMND3DNDCA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:02.378510547 (493h28m2.249593582s) [upload=433115]: Next returned nil segment Dec 26 04:57:02.378544913 (493h28m2.249627958s) [upload=433115]: Waiting for error group managing segments... Dec 26 04:57:02.758822895 (493h28m2.62990593s) [noise=true , address=145.79.227.15:4873 , piece_id=DWJPRCDVDMTNBBD7IUVMXXMOWAIEKAQSLDC7QZ56SCSUUSXUT36A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.758943015 (493h28m2.63002606s) [noise=true , address=145.79.227.41:4087 , piece_id=7FDVP5YAUBC3EQUW7BVJL4YWBBRCJJRJFHXBNPITQUSHSAUZS7BA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.763261908 (493h28m2.634344943s) [noise=true , address=145.79.227.29:4941 , piece_id=4D4F6QUU27M4SHX5EBOOZ23ZOCGJFEGD53PQYKNR5EHAOBWXDC7Q, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.764096297 (493h28m2.635179332s) [noise=true , address=145.79.227.17:4970 , piece_id=LHZJF36OFDUX3LARKFPSL5AAHHGIZ4NXNX7RE2XLC4YSOM6UKSVQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.766834852 (493h28m2.637917887s) [noise=true , address=145.79.227.42:4122 , piece_id=LZUFOCM2IQFREBOE6O5ZDBXSEMIDACH7B7YDH37TFEHT7SKEDL6Q, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.771434841 (493h28m2.642517876s) [noise=true , address=145.79.227.36:4518 , piece_id=YDL62RACOFXZBL5MNHRAM5RXYB3MGIOD5OZRNIS5B2BJUCWBH7RA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.771917460 (493h28m2.643000505s) [noise=true , address=145.79.227.19:4079 , piece_id=WE423SOQQPRWFL4XCLW3V3MT5FRPJN2XQDBOYJNQ5PSMND3DNDCA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.778998393 (493h28m2.650081428s) [noise=true , address=145.79.227.38:4978 , piece_id=REDW7YRN2GTHZP3X2XNBXYAYSZZ7RCAYGMFTKDF2NXRZRWBJCXZQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.781024683 (493h28m2.652107718s) [noise=true , address=145.79.227.46:4601 , piece_id=X64PRZMCWUULW76CGTW6KIFC45PCXZKDD6IWCR55K267OIMTFYVQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.782549526 (493h28m2.653632561s) [noise=true , address=145.79.227.35:4482 , piece_id=BAPUKYK7CIULIZW2TVAUD4GWLE66X5S3AEMCYTK2XUGC3YSNW5LA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.009543416 (493h28m2.880626461s) [noise=true , address=45.91.116.10:4082 , piece_id=FT57Q64IUDACKAYHGFG5CTYCVP6E2FQ7SFMLZFJUJKYL5SIR5UPA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.015927503 (493h28m2.887010538s) [noise=true , address=45.91.116.6:4974 , piece_id=B2E335J5GMUAMWF3NUGYMYSFEIKMIQ6HWIQI5COWLZ3XEBF4RI7A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.028581768 (493h28m2.899664813s) [noise=true , address=77.90.31.10:4519 , piece_id=CU7ZQ7UCOAEHIA4K5DWXKRIQLUTCJHTZWT6EP3FGEL62P4HXC5KA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.028609521 (493h28m2.899692557s) [seg_pos={0 0}, upload=433115]: Segment reached optimal threshold of 13 pieces. Dec 26 04:57:03.029032897 (493h28m2.900115943s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection Dec 26 04:57:03.029041354 (493h28m2.900124399s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.029625316 (493h28m2.900708361s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection Dec 26 04:57:03.029637359 (493h28m2.900720404s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.030032972 (493h28m2.901116017s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 04:57:03.030047539 (493h28m2.901130585s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.030073750 (493h28m2.901156785s) [seg_pos={0 0}, upload=433115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:57:03.030140557 (493h28m2.901223592s) [upload=433115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:57:03.030147891 (493h28m2.901230936s) [upload=433115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:57:03.326703811 (493h28m3.197786856s) [upload=433115]: Done uploading segments... Dec 26 04:57:03.326795645 (493h28m3.19787869s) [upload=433115]: Upload finished. err: <nil> Duration: 6.162907158s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.226106814s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:57:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.22613681s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 25.793202056s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:57:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 25.793229988s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.208354349s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:57:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.208472786s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [............X..X..X.X....XX.X.X.X.XX.X......X.....X.....XXX.] duration: 5.888710486s trace: https://jaeger.dev.storj.io/trace/35f5df21c5c50271 FAIL (2025-12-26 04:57:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28m0.364691594s) [upload=433115]: Uploading segments... Dec 26 04:57:00.493817967 (493h28m0.364901002s) [upload=433115]: Got next segment. Inline: false Dec 26 04:57:00.493858234 (493h28m0.364941269s) [upload=433115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:57:00.493863213 (493h28m0.364946248s) [upload=433115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:57:00.740808295 (493h28m0.61189134s) [seg_pos={0 0}, upload=433115]: Begin upload segment... Dec 26 04:57:00.740867859 (493h28m0.611950904s) [seg_pos={0 0}, upload=433115]: Done begin upload segment. Dec 26 04:57:00.740903106 (493h28m0.611986151s) [noise=true , address=77.90.31.10:4519 , piece_id=CU7ZQ7UCOAEHIA4K5DWXKRIQLUTCJHTZWT6EP3FGEL62P4HXC5KA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.740956066 (493h28m0.612039111s) [noise=true , address=145.79.227.36:4518 , piece_id=YDL62RACOFXZBL5MNHRAM5RXYB3MGIOD5OZRNIS5B2BJUCWBH7RA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.741498009 (493h28m0.612581054s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.741964597 (493h28m0.613047642s) [noise=true , address=45.91.116.6:4974 , piece_id=B2E335J5GMUAMWF3NUGYMYSFEIKMIQ6HWIQI5COWLZ3XEBF4RI7A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.742495277 (493h28m0.613578322s) [noise=true , address=145.79.227.38:4978 , piece_id=REDW7YRN2GTHZP3X2XNBXYAYSZZ7RCAYGMFTKDF2NXRZRWBJCXZQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.742922181 (493h28m0.614005226s) [noise=true , address=145.79.227.17:4970 , piece_id=LHZJF36OFDUX3LARKFPSL5AAHHGIZ4NXNX7RE2XLC4YSOM6UKSVQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.743365033 (493h28m0.614448078s) [noise=true , address=45.91.116.10:4082 , piece_id=FT57Q64IUDACKAYHGFG5CTYCVP6E2FQ7SFMLZFJUJKYL5SIR5UPA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.743851219 (493h28m0.614934264s) [noise=true , address=145.79.227.35:4482 , piece_id=BAPUKYK7CIULIZW2TVAUD4GWLE66X5S3AEMCYTK2XUGC3YSNW5LA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.745195890 (493h28m0.616278925s) [noise=true , address=145.79.227.15:4873 , piece_id=DWJPRCDVDMTNBBD7IUVMXXMOWAIEKAQSLDC7QZ56SCSUUSXUT36A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.747342197 (493h28m0.618425242s) [noise=true , address=145.79.227.29:4941 , piece_id=4D4F6QUU27M4SHX5EBOOZ23ZOCGJFEGD53PQYKNR5EHAOBWXDC7Q, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.747535324 (493h28m0.618618369s) [noise=true , address=145.79.227.46:4601 , piece_id=X64PRZMCWUULW76CGTW6KIFC45PCXZKDD6IWCR55K267OIMTFYVQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.750206982 (493h28m0.621290017s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.752203423 (493h28m0.623286458s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.754550804 (493h28m0.625633839s) [noise=true , address=145.79.227.41:4087 , piece_id=7FDVP5YAUBC3EQUW7BVJL4YWBBRCJJRJFHXBNPITQUSHSAUZS7BA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.755805773 (493h28m0.626888808s) [noise=true , address=145.79.227.42:4122 , piece_id=LZUFOCM2IQFREBOE6O5ZDBXSEMIDACH7B7YDH37TFEHT7SKEDL6Q, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.759546145 (493h28m0.63062919s) [noise=true , address=145.79.227.19:4079 , piece_id=WE423SOQQPRWFL4XCLW3V3MT5FRPJN2XQDBOYJNQ5PSMND3DNDCA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:02.378510547 (493h28m2.249593582s) [upload=433115]: Next returned nil segment Dec 26 04:57:02.378544913 (493h28m2.249627958s) [upload=433115]: Waiting for error group managing segments... Dec 26 04:57:02.758822895 (493h28m2.62990593s) [noise=true , address=145.79.227.15:4873 , piece_id=DWJPRCDVDMTNBBD7IUVMXXMOWAIEKAQSLDC7QZ56SCSUUSXUT36A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.758943015 (493h28m2.63002606s) [noise=true , address=145.79.227.41:4087 , piece_id=7FDVP5YAUBC3EQUW7BVJL4YWBBRCJJRJFHXBNPITQUSHSAUZS7BA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.763261908 (493h28m2.634344943s) [noise=true , address=145.79.227.29:4941 , piece_id=4D4F6QUU27M4SHX5EBOOZ23ZOCGJFEGD53PQYKNR5EHAOBWXDC7Q, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.764096297 (493h28m2.635179332s) [noise=true , address=145.79.227.17:4970 , piece_id=LHZJF36OFDUX3LARKFPSL5AAHHGIZ4NXNX7RE2XLC4YSOM6UKSVQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.766834852 (493h28m2.637917887s) [noise=true , address=145.79.227.42:4122 , piece_id=LZUFOCM2IQFREBOE6O5ZDBXSEMIDACH7B7YDH37TFEHT7SKEDL6Q, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.771434841 (493h28m2.642517876s) [noise=true , address=145.79.227.36:4518 , piece_id=YDL62RACOFXZBL5MNHRAM5RXYB3MGIOD5OZRNIS5B2BJUCWBH7RA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.771917460 (493h28m2.643000505s) [noise=true , address=145.79.227.19:4079 , piece_id=WE423SOQQPRWFL4XCLW3V3MT5FRPJN2XQDBOYJNQ5PSMND3DNDCA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.778998393 (493h28m2.650081428s) [noise=true , address=145.79.227.38:4978 , piece_id=REDW7YRN2GTHZP3X2XNBXYAYSZZ7RCAYGMFTKDF2NXRZRWBJCXZQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.781024683 (493h28m2.652107718s) [noise=true , address=145.79.227.46:4601 , piece_id=X64PRZMCWUULW76CGTW6KIFC45PCXZKDD6IWCR55K267OIMTFYVQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.782549526 (493h28m2.653632561s) [noise=true , address=145.79.227.35:4482 , piece_id=BAPUKYK7CIULIZW2TVAUD4GWLE66X5S3AEMCYTK2XUGC3YSNW5LA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.009543416 (493h28m2.880626461s) [noise=true , address=45.91.116.10:4082 , piece_id=FT57Q64IUDACKAYHGFG5CTYCVP6E2FQ7SFMLZFJUJKYL5SIR5UPA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.015927503 (493h28m2.887010538s) [noise=true , address=45.91.116.6:4974 , piece_id=B2E335J5GMUAMWF3NUGYMYSFEIKMIQ6HWIQI5COWLZ3XEBF4RI7A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.028581768 (493h28m2.899664813s) [noise=true , address=77.90.31.10:4519 , piece_id=CU7ZQ7UCOAEHIA4K5DWXKRIQLUTCJHTZWT6EP3FGEL62P4HXC5KA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.028609521 (493h28m2.899692557s) [seg_pos={0 0}, upload=433115]: Segment reached optimal threshold of 13 pieces. Dec 26 04:57:03.029032897 (493h28m2.900115943s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection Dec 26 04:57:03.029041354 (493h28m2.900124399s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.029625316 (493h28m2.900708361s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection Dec 26 04:57:03.029637359 (493h28m2.900720404s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.030032972 (493h28m2.901116017s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 04:57:03.030047539 (493h28m2.901130585s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.030073750 (493h28m2.901156785s) [seg_pos={0 0}, upload=433115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:57:03.030140557 (493h28m2.901223592s) [upload=433115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:57:03.030147891 (493h28m2.901230936s) [upload=433115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:57:03.326703811 (493h28m3.197786856s) [upload=433115]: Done uploading segments... Dec 26 04:57:03.326795645 (493h28m3.19787869s) [upload=433115]: Upload finished. err: <nil> Duration: 5.888724041s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.............X..X..X.X....XX.X.X.X.XX.X......X.....X.....XXX] duration: 30.002843288s trace: https://jaeger.dev.storj.io/trace/35f5df21c5c50271 FAIL (2025-12-26 04:57:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28m0.364691594s) [upload=433115]: Uploading segments... Dec 26 04:57:00.493817967 (493h28m0.364901002s) [upload=433115]: Got next segment. Inline: false Dec 26 04:57:00.493858234 (493h28m0.364941269s) [upload=433115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:57:00.493863213 (493h28m0.364946248s) [upload=433115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:57:00.740808295 (493h28m0.61189134s) [seg_pos={0 0}, upload=433115]: Begin upload segment... Dec 26 04:57:00.740867859 (493h28m0.611950904s) [seg_pos={0 0}, upload=433115]: Done begin upload segment. Dec 26 04:57:00.740903106 (493h28m0.611986151s) [noise=true , address=77.90.31.10:4519 , piece_id=CU7ZQ7UCOAEHIA4K5DWXKRIQLUTCJHTZWT6EP3FGEL62P4HXC5KA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.740956066 (493h28m0.612039111s) [noise=true , address=145.79.227.36:4518 , piece_id=YDL62RACOFXZBL5MNHRAM5RXYB3MGIOD5OZRNIS5B2BJUCWBH7RA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.741498009 (493h28m0.612581054s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.741964597 (493h28m0.613047642s) [noise=true , address=45.91.116.6:4974 , piece_id=B2E335J5GMUAMWF3NUGYMYSFEIKMIQ6HWIQI5COWLZ3XEBF4RI7A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.742495277 (493h28m0.613578322s) [noise=true , address=145.79.227.38:4978 , piece_id=REDW7YRN2GTHZP3X2XNBXYAYSZZ7RCAYGMFTKDF2NXRZRWBJCXZQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.742922181 (493h28m0.614005226s) [noise=true , address=145.79.227.17:4970 , piece_id=LHZJF36OFDUX3LARKFPSL5AAHHGIZ4NXNX7RE2XLC4YSOM6UKSVQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.743365033 (493h28m0.614448078s) [noise=true , address=45.91.116.10:4082 , piece_id=FT57Q64IUDACKAYHGFG5CTYCVP6E2FQ7SFMLZFJUJKYL5SIR5UPA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.743851219 (493h28m0.614934264s) [noise=true , address=145.79.227.35:4482 , piece_id=BAPUKYK7CIULIZW2TVAUD4GWLE66X5S3AEMCYTK2XUGC3YSNW5LA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.745195890 (493h28m0.616278925s) [noise=true , address=145.79.227.15:4873 , piece_id=DWJPRCDVDMTNBBD7IUVMXXMOWAIEKAQSLDC7QZ56SCSUUSXUT36A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.747342197 (493h28m0.618425242s) [noise=true , address=145.79.227.29:4941 , piece_id=4D4F6QUU27M4SHX5EBOOZ23ZOCGJFEGD53PQYKNR5EHAOBWXDC7Q, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.747535324 (493h28m0.618618369s) [noise=true , address=145.79.227.46:4601 , piece_id=X64PRZMCWUULW76CGTW6KIFC45PCXZKDD6IWCR55K267OIMTFYVQ, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.750206982 (493h28m0.621290017s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.752203423 (493h28m0.623286458s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.754550804 (493h28m0.625633839s) [noise=true , address=145.79.227.41:4087 , piece_id=7FDVP5YAUBC3EQUW7BVJL4YWBBRCJJRJFHXBNPITQUSHSAUZS7BA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.755805773 (493h28m0.626888808s) [noise=true , address=145.79.227.42:4122 , piece_id=LZUFOCM2IQFREBOE6O5ZDBXSEMIDACH7B7YDH37TFEHT7SKEDL6Q, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:00.759546145 (493h28m0.63062919s) [noise=true , address=145.79.227.19:4079 , piece_id=WE423SOQQPRWFL4XCLW3V3MT5FRPJN2XQDBOYJNQ5PSMND3DNDCA, seg_pos={0 0}, upload=433115]: Uploading piece... Dec 26 04:57:02.378510547 (493h28m2.249593582s) [upload=433115]: Next returned nil segment Dec 26 04:57:02.378544913 (493h28m2.249627958s) [upload=433115]: Waiting for error group managing segments... Dec 26 04:57:02.758822895 (493h28m2.62990593s) [noise=true , address=145.79.227.15:4873 , piece_id=DWJPRCDVDMTNBBD7IUVMXXMOWAIEKAQSLDC7QZ56SCSUUSXUT36A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.758943015 (493h28m2.63002606s) [noise=true , address=145.79.227.41:4087 , piece_id=7FDVP5YAUBC3EQUW7BVJL4YWBBRCJJRJFHXBNPITQUSHSAUZS7BA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.763261908 (493h28m2.634344943s) [noise=true , address=145.79.227.29:4941 , piece_id=4D4F6QUU27M4SHX5EBOOZ23ZOCGJFEGD53PQYKNR5EHAOBWXDC7Q, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.764096297 (493h28m2.635179332s) [noise=true , address=145.79.227.17:4970 , piece_id=LHZJF36OFDUX3LARKFPSL5AAHHGIZ4NXNX7RE2XLC4YSOM6UKSVQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.766834852 (493h28m2.637917887s) [noise=true , address=145.79.227.42:4122 , piece_id=LZUFOCM2IQFREBOE6O5ZDBXSEMIDACH7B7YDH37TFEHT7SKEDL6Q, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.771434841 (493h28m2.642517876s) [noise=true , address=145.79.227.36:4518 , piece_id=YDL62RACOFXZBL5MNHRAM5RXYB3MGIOD5OZRNIS5B2BJUCWBH7RA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.771917460 (493h28m2.643000505s) [noise=true , address=145.79.227.19:4079 , piece_id=WE423SOQQPRWFL4XCLW3V3MT5FRPJN2XQDBOYJNQ5PSMND3DNDCA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.778998393 (493h28m2.650081428s) [noise=true , address=145.79.227.38:4978 , piece_id=REDW7YRN2GTHZP3X2XNBXYAYSZZ7RCAYGMFTKDF2NXRZRWBJCXZQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.781024683 (493h28m2.652107718s) [noise=true , address=145.79.227.46:4601 , piece_id=X64PRZMCWUULW76CGTW6KIFC45PCXZKDD6IWCR55K267OIMTFYVQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:02.782549526 (493h28m2.653632561s) [noise=true , address=145.79.227.35:4482 , piece_id=BAPUKYK7CIULIZW2TVAUD4GWLE66X5S3AEMCYTK2XUGC3YSNW5LA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.009543416 (493h28m2.880626461s) [noise=true , address=45.91.116.10:4082 , piece_id=FT57Q64IUDACKAYHGFG5CTYCVP6E2FQ7SFMLZFJUJKYL5SIR5UPA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.015927503 (493h28m2.887010538s) [noise=true , address=45.91.116.6:4974 , piece_id=B2E335J5GMUAMWF3NUGYMYSFEIKMIQ6HWIQI5COWLZ3XEBF4RI7A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.028581768 (493h28m2.899664813s) [noise=true , address=77.90.31.10:4519 , piece_id=CU7ZQ7UCOAEHIA4K5DWXKRIQLUTCJHTZWT6EP3FGEL62P4HXC5KA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: <nil> Dec 26 04:57:03.028609521 (493h28m2.899692557s) [seg_pos={0 0}, upload=433115]: Segment reached optimal threshold of 13 pieces. Dec 26 04:57:03.029032897 (493h28m2.900115943s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58818->45.91.116.13:4156: use of closed network connection Dec 26 04:57:03.029041354 (493h28m2.900124399s) [noise=true , address=45.91.116.13:4156 , piece_id=HO3VEXEEV4QZWFN6C472MPHOMVAFH4QR5BCM7WJCEC27GVSBNA4A, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.029625316 (493h28m2.900708361s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45776->45.91.116.11:4117: use of closed network connection Dec 26 04:57:03.029637359 (493h28m2.900720404s) [noise=true , address=45.91.116.11:4117 , piece_id=6QCQA3UQU7FFPSEDHLNVZ57D6ZOSABATRBJ3F5YXTTN4G6KQTHGQ, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.030032972 (493h28m2.901116017s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 04:57:03.030047539 (493h28m2.901130585s) [noise=true , address=45.91.116.12:4225 , piece_id=XRCYBOUVH6I6ALBO3LTXL3HLT6F2Q3JYCWLQHJFMFI2RIJTUXCLA, seg_pos={0 0}, upload=433115]: Long tail cancellation due to optimal threshold Dec 26 04:57:03.030073750 (493h28m2.901156785s) [seg_pos={0 0}, upload=433115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:57:03.030140557 (493h28m2.901223592s) [upload=433115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:57:03.030147891 (493h28m2.901230936s) [upload=433115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:57:03.326703811 (493h28m3.197786856s) [upload=433115]: Done uploading segments... Dec 26 04:57:03.326795645 (493h28m3.19787869s) [upload=433115]: Upload finished. err: <nil> Duration: 30.002866031s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 28.834703761s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:56:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 28.834730852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..............X..X..X.X....XX.X.X.X.XX.X......X.....X.....XX] duration: 30.003015927s trace: https://jaeger.dev.storj.io/trace/1658b08a2503924b FAIL (2025-12-26 04:56:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: APKXP634SPXA75ZDDQT4HTPUOVZTLMQJ3NUX4ZQOLDABR5E6375Q) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: FSZIMXO2WDE7SH5KG4SUORVISFNKP6R3DLC3PTKKDVOLAUXIMHYA) context deadline exceeded; piecestore: (Node ID: 12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT, Piece ID: VGYVGKCHW7BMO7ZSPR5XOYD55BEMYQO7US4MKG3M3I2IWMN4TMYQ) context deadline exceeded ======LOGS====== Dec 26 04:56:21.947829313 (493h27m21.818912368s) [upload=433107]: Starting upload Dec 26 04:56:21.947932319 (493h27m21.819015354s) [upload=433107]: Done starting upload Dec 26 04:56:21.948020928 (493h27m21.819103993s) [upload=433107]: Uploading segments... Dec 26 04:56:21.948207813 (493h27m21.819290859s) [upload=433107]: Got next segment. Inline: false Dec 26 04:56:21.948252238 (493h27m21.819335283s) [upload=433107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:56:21.948257839 (493h27m21.819340884s) [upload=433107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:56:23.209272025 (493h27m23.08035506s) [seg_pos={0 0}, upload=433107]: Begin upload segment... Dec 26 04:56:23.209323152 (493h27m23.080406187s) [seg_pos={0 0}, upload=433107]: Done begin upload segment. Dec 26 04:56:23.209344303 (493h27m23.080427348s) [noise=true , address=145.79.227.39:4014 , piece_id=ZZQFZPC3TO2RHU7QYLRRJDZW2SHQ66HWMW22JLRGMRU42TWEBXYA, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.209354973 (493h27m23.080438018s) [noise=true , address=145.79.227.15:4876 , piece_id=4MX5DFT6RFENSRS2JTPQZDLLIC7KCXQR5Q4QIWZO624ZNE5NMQBA, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.210345379 (493h27m23.081428414s) [noise=true , address=77.90.31.19:4844 , piece_id=OSPIVXEZGS7EUE5RTXWNMZHJGGP5T2UPF2Y3NMNFA2PY6LZMIK3A, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.210476077 (493h27m23.081559122s) [noise=true , address=145.79.227.4:4187 , piece_id=APKXP634SPXA75ZDDQT4HTPUOVZTLMQJ3NUX4ZQOLDABR5E6375Q, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.210845170 (493h27m23.081928216s) [noise=true , address=45.91.116.14:4192 , piece_id=2UGD2AGG3LDU2OFHBOLGJZ44RF5JZH63BOX2IRHXXUQGGEKZV6AQ, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.210963095 (493h27m23.08204614s) [noise=true , address=145.79.227.29:4941 , piece_id=ZOP6L62EABW2PLQC5365EGF5R3JFY2RGH2PDN3I7CN7Z5WFYDCGA, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.211262385 (493h27m23.08234542s) [noise=true , address=145.79.227.45:4231 , piece_id=SZ5NSB6JXIX4FMEOEXI2LJJJLKF772GLOY2FLBX5REEBKUG25TQQ, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.211657898 (493h27m23.082740933s) [noise=true , address=145.79.227.49:4337 , piece_id=FSZIMXO2WDE7SH5KG4SUORVISFNKP6R3DLC3PTKKDVOLAUXIMHYA, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.212048001 (493h27m23.083131056s) [noise=true , address=145.79.227.36:4517 , piece_id=V2XM32TLFXUSWF62RWW6N5KKAEN5ZLXHD2FZRK5URMWCXMCPEMIA, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.212396014 (493h27m23.083479049s) [noise=true , address=145.79.227.30:4762 , piece_id=K4J6VWK3VOUJUAIXAU5GPAPUK3ZWDH7ST5BSQFCWTVAEC5JDIXSQ, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.212917216 (493h27m23.084000262s) [noise=true , address=145.79.227.17:4970 , piece_id=UNIIUQMFWCHBTSARNK4YDLX6XK3HBMMIZC375FATBZHP7U4KDHFA, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.213425133 (493h27m23.084508178s) [noise=true , address=145.79.227.50:4374 , piece_id=KAR5GOAW4ISOHYXKUHA7ZTS77PRVDBKCUF7MV4TRNGN6GRL5QJIA, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.213853209 (493h27m23.084936244s) [noise=true , address=145.79.227.46:4602 , piece_id=SCS6AUMI6EU75SBFF5MALT5IPVUIM7RC6TPBLC44VQPSHGRNJBPQ, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.214209647 (493h27m23.085292692s) [noise=true , address=145.79.227.48:4303 , piece_id=VGYVGKCHW7BMO7ZSPR5XOYD55BEMYQO7US4MKG3M3I2IWMN4TMYQ, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.214486324 (493h27m23.085569369s) [noise=true , address=45.91.116.7:4012 , piece_id=2PBPOXYNVNCBWCGEGSVA6BD4NUWDSUIKM4TSJSDEVZ7WHPS2C7YA, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:23.215033185 (493h27m23.08611622s) [noise=true , address=145.79.227.14:4838 , piece_id=USPPO6EZMX3CTRXAYFL7MPXXNDNUW36OGABWAAQE2JBSLKDVXRXA, seg_pos={0 0}, upload=433107]: Uploading piece... Dec 26 04:56:24.986412124 (493h27m24.857495169s) [upload=433107]: Next returned nil segment Dec 26 04:56:24.986424148 (493h27m24.857507183s) [upload=433107]: Waiting for error group managing segments... Dec 26 04:56:25.364210295 (493h27m25.23529333s) [noise=true , address=145.79.227.49:4337 , piece_id=FSZIMXO2WDE7SH5KG4SUORVISFNKP6R3DLC3PTKKDVOLAUXIMHYA, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:25.399667021 (493h27m25.270750056s) [noise=true , address=145.79.227.15:4876 , piece_id=4MX5DFT6RFENSRS2JTPQZDLLIC7KCXQR5Q4QIWZO624ZNE5NMQBA, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:25.411345099 (493h27m25.282428134s) [noise=true , address=145.79.227.29:4941 , piece_id=ZOP6L62EABW2PLQC5365EGF5R3JFY2RGH2PDN3I7CN7Z5WFYDCGA, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:25.415812737 (493h27m25.286895772s) [noise=true , address=145.79.227.4:4187 , piece_id=APKXP634SPXA75ZDDQT4HTPUOVZTLMQJ3NUX4ZQOLDABR5E6375Q, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:25.422754304 (493h27m25.293837349s) [noise=true , address=145.79.227.17:4970 , piece_id=UNIIUQMFWCHBTSARNK4YDLX6XK3HBMMIZC375FATBZHP7U4KDHFA, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:25.424947130 (493h27m25.296030175s) [noise=true , address=145.79.227.36:4517 , piece_id=V2XM32TLFXUSWF62RWW6N5KKAEN5ZLXHD2FZRK5URMWCXMCPEMIA, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:25.431445324 (493h27m25.302528359s) [noise=true , address=145.79.227.39:4014 , piece_id=ZZQFZPC3TO2RHU7QYLRRJDZW2SHQ66HWMW22JLRGMRU42TWEBXYA, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:25.455233962 (493h27m25.326316997s) [noise=true , address=45.91.116.14:4192 , piece_id=2UGD2AGG3LDU2OFHBOLGJZ44RF5JZH63BOX2IRHXXUQGGEKZV6AQ, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:25.496405220 (493h27m25.367488255s) [noise=true , address=145.79.227.50:4374 , piece_id=KAR5GOAW4ISOHYXKUHA7ZTS77PRVDBKCUF7MV4TRNGN6GRL5QJIA, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:25.534638875 (493h27m25.40572192s) [noise=true , address=77.90.31.19:4844 , piece_id=OSPIVXEZGS7EUE5RTXWNMZHJGGP5T2UPF2Y3NMNFA2PY6LZMIK3A, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:25.810430625 (493h27m25.68151367s) [noise=true , address=45.91.116.7:4012 , piece_id=2PBPOXYNVNCBWCGEGSVA6BD4NUWDSUIKM4TSJSDEVZ7WHPS2C7YA, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:26.071644357 (493h27m25.942727392s) [noise=true , address=145.79.227.48:4303 , piece_id=VGYVGKCHW7BMO7ZSPR5XOYD55BEMYQO7US4MKG3M3I2IWMN4TMYQ, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:26.097373580 (493h27m25.968456625s) [noise=true , address=145.79.227.46:4602 , piece_id=SCS6AUMI6EU75SBFF5MALT5IPVUIM7RC6TPBLC44VQPSHGRNJBPQ, seg_pos={0 0}, upload=433107]: Done uploading piece. err: <nil> Dec 26 04:56:26.097389972 (493h27m25.968473017s) [seg_pos={0 0}, upload=433107]: Segment reached optimal threshold of 13 pieces. Dec 26 04:56:26.097573651 (493h27m25.968656686s) [noise=true , address=145.79.227.45:4231 , piece_id=SZ5NSB6JXIX4FMEOEXI2LJJJLKF772GLOY2FLBX5REEBKUG25TQQ, seg_pos={0 0}, upload=433107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4): protocol: expected piece hash; context canceled; EOF Dec 26 04:56:26.097582027 (493h27m25.968665062s) [noise=true , address=145.79.227.45:4231 , piece_id=SZ5NSB6JXIX4FMEOEXI2LJJJLKF772GLOY2FLBX5REEBKUG25TQQ, seg_pos={0 0}, upload=433107]: Long tail cancellation due to optimal threshold Dec 26 04:56:26.097703669 (493h27m25.968786704s) [noise=true , address=145.79.227.14:4838 , piece_id=USPPO6EZMX3CTRXAYFL7MPXXNDNUW36OGABWAAQE2JBSLKDVXRXA, seg_pos={0 0}, upload=433107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Dec 26 04:56:26.097720701 (493h27m25.968803736s) [noise=true , address=145.79.227.14:4838 , piece_id=USPPO6EZMX3CTRXAYFL7MPXXNDNUW36OGABWAAQE2JBSLKDVXRXA, seg_pos={0 0}, upload=433107]: Long tail cancellation due to optimal threshold Dec 26 04:56:26.097898179 (493h27m25.968981214s) [noise=true , address=145.79.227.30:4762 , piece_id=K4J6VWK3VOUJUAIXAU5GPAPUK3ZWDH7ST5BSQFCWTVAEC5JDIXSQ, seg_pos={0 0}, upload=433107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 04:56:26.097906235 (493h27m25.96898927s) [noise=true , address=145.79.227.30:4762 , piece_id=K4J6VWK3VOUJUAIXAU5GPAPUK3ZWDH7ST5BSQFCWTVAEC5JDIXSQ, seg_pos={0 0}, upload=433107]: Long tail cancellation due to optimal threshold Dec 26 04:56:26.097923848 (493h27m25.969006883s) [seg_pos={0 0}, upload=433107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:56:26.098052814 (493h27m25.969135849s) [upload=433107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:56:26.098059366 (493h27m25.969142411s) [upload=433107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:56:26.410749939 (493h27m26.281832975s) [upload=433107]: Done uploading segments... Dec 26 04:56:26.410770218 (493h27m26.281853253s) [upload=433107]: Upload finished. err: <nil> Duration: 30.003026507s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.21930851s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:56:42 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.219334831s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...............X..X..X.X....XX.X.X.X.XX.X......X.....X.....X] duration: 30.002988708s trace: https://jaeger.dev.storj.io/trace/4147d3c6ae4642f0 FAIL (2025-12-26 04:56:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: CQSTNTEBCB7YNFQIU6JWHUEDRA6FHTRKVLEVGIFXU7SUK4GWHPUA) context deadline exceeded; piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: FVPDCNKSJ7VCYN3Z42YULB6M4H74OLMPQC3RRUYYOAYYNG5L76JQ) context deadline exceeded; piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: SCUEUUOMCD5SDAOWVW26YQYP76UUHE6QT45FC3K64MVK6BADW5NA) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: OG4KKD4CVQGWZDYVUZPJJXP3UHSOYTH2QP556MYWHASPKFQZLIWA) context deadline exceeded; piecestore: (Node ID: 1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, Piece ID: GQJHRWAILUNZGVWYRPDXFYPNJDAK2DBFH4DIWJ5LLTO4NBF2RIYQ) context deadline exceeded ======LOGS====== 5:50.242835285 (493h26m50.11391833s) [upload=433099]: Starting upload Dec 26 04:55:50.242914946 (493h26m50.113997991s) [upload=433099]: Done starting upload Dec 26 04:55:50.242979529 (493h26m50.114062574s) [upload=433099]: Uploading segments... Dec 26 04:55:50.243111000 (493h26m50.114194035s) [upload=433099]: Got next segment. Inline: false Dec 26 04:55:50.243159753 (493h26m50.114242788s) [upload=433099]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:55:50.243164613 (493h26m50.114247648s) [upload=433099]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:55:52.214651273 (493h26m52.085734318s) [seg_pos={0 0}, upload=433099]: Begin upload segment... Dec 26 04:55:52.214719002 (493h26m52.085802077s) [seg_pos={0 0}, upload=433099]: Done begin upload segment. Dec 26 04:55:52.214762495 (493h26m52.08584556s) [noise=true , address=145.79.227.42:4123 , piece_id=LFN4VRPVZT73W5ZYN5BPTJGH5LZUNQ7ZV5EOU25MXIMGEHVYSCIQ, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.217965884 (493h26m52.089048919s) [noise=true , address=77.90.31.21:4877 , piece_id=IPIB5SMI3KLBPVB7EHZ7XAHTRD2W3E5KYKR6UXLRDFIH42ROP3WA, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.218465286 (493h26m52.089548321s) [noise=true , address=145.79.227.15:4873 , piece_id=P2BJBOC4TQBED5EXEHN6X7XNXCMLQHDW767YDHGARBJI7NQSEGTQ, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.220004296 (493h26m52.091087341s) [noise=true , address=145.79.227.11:4731 , piece_id=REISJPPQR5BGP2RFWI3VMR2RRZONCHHXPZGPMSY4YEPCXW6FLNLA, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.220493949 (493h26m52.091576984s) [noise=true , address=45.91.116.11:4119 , piece_id=FFQ6N3CD2EO3LWFS7KN3B7XYW3YKO2EETF47ZWUOURACSIKPWMHQ, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.223152791 (493h26m52.094235836s) [noise=true , address=145.79.227.2:4115 , piece_id=FVPDCNKSJ7VCYN3Z42YULB6M4H74OLMPQC3RRUYYOAYYNG5L76JQ, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.223502768 (493h26m52.094585814s) [noise=true , address=45.91.116.10:4083 , piece_id=GJRW53TEXFL3B4PZU24A3LXQA4TSU3XHZEDPUXPSVRDAO3INGJ7Q, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.225606113 (493h26m52.096689158s) [noise=true , address=145.79.227.17:4971 , piece_id=6FLFTI7GY4PCUICGVQ6Y2H43B75RNBECDW62G2CJALXQVXO3X2EQ, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.226127185 (493h26m52.09721023s) [noise=true , address=145.79.227.50:4373 , piece_id=HLLKOU4DZSNNVHCUIVZKHVHLHKJO7E3722JG2WCMSYTJBC3GQZIA, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.227688048 (493h26m52.098771103s) [noise=true , address=145.79.227.16:4912 , piece_id=3GLVV7HCVZBASU7RSQEQAWJQ4ZJCYW4I7KZW4R6DGSEQ6DNWN2YA, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.228704002 (493h26m52.099787037s) [noise=true , address=145.79.227.35:4482 , piece_id=SCUEUUOMCD5SDAOWVW26YQYP76UUHE6QT45FC3K64MVK6BADW5NA, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.229210696 (493h26m52.100293731s) [noise=true , address=145.79.227.7:4294 , piece_id=OG4KKD4CVQGWZDYVUZPJJXP3UHSOYTH2QP556MYWHASPKFQZLIWA, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.230030388 (493h26m52.101113433s) [noise=true , address=145.79.227.51:4410 , piece_id=F33VOR25BICKURIYLCXVNXTNKK3EG44JJCMJKUYZ4LBXCRMLCXFQ, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.231146052 (493h26m52.102229097s) [noise=true , address=145.79.227.18:4007 , piece_id=SNFRHXXY5B3X6IFSYFIGJ3MHOTDN6LP4UOINSXL5BY65OTIY6FLA, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.235368864 (493h26m52.106451919s) [noise=true , address=145.79.227.41:4086 , piece_id=GQJHRWAILUNZGVWYRPDXFYPNJDAK2DBFH4DIWJ5LLTO4NBF2RIYQ, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:52.237414910 (493h26m52.108497955s) [noise=true , address=45.91.116.5:4300 , piece_id=CQSTNTEBCB7YNFQIU6JWHUEDRA6FHTRKVLEVGIFXU7SUK4GWHPUA, seg_pos={0 0}, upload=433099]: Uploading piece... Dec 26 04:55:53.406224612 (493h26m53.277307657s) [noise=true , address=77.90.31.21:4877 , piece_id=IPIB5SMI3KLBPVB7EHZ7XAHTRD2W3E5KYKR6UXLRDFIH42ROP3WA, seg_pos={0 0}, upload=433099]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 04:55:54.086251685 (493h26m53.95733473s) [upload=433099]: Next returned nil segment Dec 26 04:55:54.086269960 (493h26m53.957352995s) [upload=433099]: Waiting for error group managing segments... Dec 26 04:55:54.387887589 (493h26m54.258970634s) [noise=true , address=145.79.227.15:4873 , piece_id=P2BJBOC4TQBED5EXEHN6X7XNXCMLQHDW767YDHGARBJI7NQSEGTQ, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.391976254 (493h26m54.263059289s) [noise=true , address=145.79.227.41:4086 , piece_id=GQJHRWAILUNZGVWYRPDXFYPNJDAK2DBFH4DIWJ5LLTO4NBF2RIYQ, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.407868917 (493h26m54.278951962s) [noise=true , address=145.79.227.35:4482 , piece_id=SCUEUUOMCD5SDAOWVW26YQYP76UUHE6QT45FC3K64MVK6BADW5NA, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.420391714 (493h26m54.291474749s) [noise=true , address=145.79.227.51:4410 , piece_id=F33VOR25BICKURIYLCXVNXTNKK3EG44JJCMJKUYZ4LBXCRMLCXFQ, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.420846449 (493h26m54.291929495s) [noise=true , address=145.79.227.2:4115 , piece_id=FVPDCNKSJ7VCYN3Z42YULB6M4H74OLMPQC3RRUYYOAYYNG5L76JQ, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.420944135 (493h26m54.29202717s) [noise=true , address=145.79.227.11:4731 , piece_id=REISJPPQR5BGP2RFWI3VMR2RRZONCHHXPZGPMSY4YEPCXW6FLNLA, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.438569217 (493h26m54.309652252s) [noise=true , address=145.79.227.42:4123 , piece_id=LFN4VRPVZT73W5ZYN5BPTJGH5LZUNQ7ZV5EOU25MXIMGEHVYSCIQ, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.456947342 (493h26m54.328030387s) [noise=true , address=45.91.116.5:4300 , piece_id=CQSTNTEBCB7YNFQIU6JWHUEDRA6FHTRKVLEVGIFXU7SUK4GWHPUA, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.461400011 (493h26m54.332483047s) [noise=true , address=145.79.227.7:4294 , piece_id=OG4KKD4CVQGWZDYVUZPJJXP3UHSOYTH2QP556MYWHASPKFQZLIWA, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.485372339 (493h26m54.356455374s) [noise=true , address=45.91.116.10:4083 , piece_id=GJRW53TEXFL3B4PZU24A3LXQA4TSU3XHZEDPUXPSVRDAO3INGJ7Q, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.517411508 (493h26m54.388494543s) [noise=true , address=145.79.227.16:4912 , piece_id=3GLVV7HCVZBASU7RSQEQAWJQ4ZJCYW4I7KZW4R6DGSEQ6DNWN2YA, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.577042316 (493h26m54.448125351s) [noise=true , address=45.91.116.11:4119 , piece_id=FFQ6N3CD2EO3LWFS7KN3B7XYW3YKO2EETF47ZWUOURACSIKPWMHQ, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.718726771 (493h26m54.589809806s) [noise=true , address=145.79.227.18:4007 , piece_id=SNFRHXXY5B3X6IFSYFIGJ3MHOTDN6LP4UOINSXL5BY65OTIY6FLA, seg_pos={0 0}, upload=433099]: Done uploading piece. err: <nil> Dec 26 04:55:54.718749244 (493h26m54.589832279s) [seg_pos={0 0}, upload=433099]: Segment reached optimal threshold of 13 pieces. Dec 26 04:55:54.718975084 (493h26m54.590058129s) [noise=true , address=145.79.227.50:4373 , piece_id=HLLKOU4DZSNNVHCUIVZKHVHLHKJO7E3722JG2WCMSYTJBC3GQZIA, seg_pos={0 0}, upload=433099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:34194->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:34194->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34194->145.79.227.50:4373: use of closed network connection Dec 26 04:55:54.718985224 (493h26m54.590068259s) [noise=true , address=145.79.227.50:4373 , piece_id=HLLKOU4DZSNNVHCUIVZKHVHLHKJO7E3722JG2WCMSYTJBC3GQZIA, seg_pos={0 0}, upload=433099]: Long tail cancellation due to optimal threshold Dec 26 04:55:54.719141932 (493h26m54.590224977s) [noise=true , address=145.79.227.17:4971 , piece_id=6FLFTI7GY4PCUICGVQ6Y2H43B75RNBECDW62G2CJALXQVXO3X2EQ, seg_pos={0 0}, upload=433099]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:35684->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:35684->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35684->145.79.227.17:4971: use of closed network connection Dec 26 04:55:54.719150107 (493h26m54.590233152s) [noise=true , address=145.79.227.17:4971 , piece_id=6FLFTI7GY4PCUICGVQ6Y2H43B75RNBECDW62G2CJALXQVXO3X2EQ, seg_pos={0 0}, upload=433099]: Long tail cancellation due to optimal threshold Dec 26 04:55:54.719170456 (493h26m54.590253491s) [seg_pos={0 0}, upload=433099]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:55:54.719245609 (493h26m54.590328644s) [upload=433099]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:55:54.719251720 (493h26m54.590334755s) [upload=433099]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:55:55.032578496 (493h26m54.903661541s) [upload=433099]: Done uploading segments... Dec 26 04:55:55.032597382 (493h26m54.903680427s) [upload=433099]: Upload finished. err: <nil> Duration: 30.003013697s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.018838508s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:56:05 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.018874437s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 26.584097474s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:55:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 26.584138694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.303626729s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:55:42 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.303653089s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [................X..X..X.X....XX.X.X.X.XX.X......X.....X.....] duration: 5.971938919s trace: https://jaeger.dev.storj.io/trace/47b8400ca92cf8ee FAIL (2025-12-26 04:55:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, Piece ID: 2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: 37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: 32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ) context deadline exceeded ======LOGS====== h22m1.325682255s) [upload=433027]: Uploading segments... Dec 26 04:51:01.454784964 (493h22m1.325868009s) [upload=433027]: Got next segment. Inline: false Dec 26 04:51:01.454842182 (493h22m1.325925227s) [upload=433027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:51:01.454848625 (493h22m1.32593167s) [upload=433027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:51:01.716256741 (493h22m1.587339786s) [seg_pos={0 0}, upload=433027]: Begin upload segment... Dec 26 04:51:01.716321104 (493h22m1.587404149s) [seg_pos={0 0}, upload=433027]: Done begin upload segment. Dec 26 04:51:01.716350710 (493h22m1.587433755s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.716416826 (493h22m1.587499871s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.717137017 (493h22m1.588220062s) [noise=true , address=145.79.227.4:4185 , piece_id=EJ2CPERUCGKDIVIFA3JB3D46QLMZIUKYCLWNOVZ4WZ6QU5DPH4FA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.717275030 (493h22m1.588358075s) [noise=true , address=145.79.227.38:4977 , piece_id=DAIFVBQ3USULKMUBMMMQ6QEM6C3FYWR26A5YUSBUQW2SRROCOJAQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718006162 (493h22m1.589089207s) [noise=true , address=145.79.227.43:4157 , piece_id=37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718273662 (493h22m1.589356707s) [noise=true , address=45.91.116.13:4154 , piece_id=LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718632625 (493h22m1.58971566s) [noise=true , address=145.79.227.5:4223 , piece_id=32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.719228028 (493h22m1.590311074s) [noise=true , address=145.79.227.49:4339 , piece_id=L2F3BLLRK5J5XZZL2RXRINDPVDOF3LSAE3IELCY3VSFC54VN6GIA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.720480954 (493h22m1.591563999s) [noise=true , address=145.79.227.36:4517 , piece_id=W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.721203200 (493h22m1.592286245s) [noise=true , address=145.79.227.50:4374 , piece_id=65RLVWP4VY3P5DZX3XNFR34RQIGMKP2T4T7MD2Y6IFVUS3W2VKXQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.721530704 (493h22m1.592613739s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.722180451 (493h22m1.593263496s) [noise=true , address=45.91.116.9:4263 , piece_id=JBCZMTOD6NRXPBN3SXZMR7IE27VAUNSB2IJK4X3RJ7YJFRWWVYQQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.723072510 (493h22m1.594155545s) [noise=true , address=145.79.227.11:4730 , piece_id=CEK3H5R4ZEBZ4YEHSXUYE5YHZ5DNNZKIR3Z3V7EXQ4YKNJ3AKKZA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.723594634 (493h22m1.594677679s) [noise=true , address=145.79.227.51:4411 , piece_id=HANFRVC6P3EODDP3M7NMQD5VHXHB4TDVUCWXOB6JB4HCLSFS6HLA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.724105607 (493h22m1.595188652s) [noise=true , address=77.90.31.11:4556 , piece_id=EXB3WOSS4NUNSM2SPREQVSTD73RP22LY7CO462DXH5MYDAWF7WFA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.724578727 (493h22m1.595661772s) [noise=true , address=145.79.227.6:4257 , piece_id=2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:03.471291472 (493h22m3.342374517s) [upload=433027]: Next returned nil segment Dec 26 04:51:03.471307903 (493h22m3.342390938s) [upload=433027]: Waiting for error group managing segments... Dec 26 04:51:03.881101321 (493h22m3.752184366s) [noise=true , address=145.79.227.51:4411 , piece_id=HANFRVC6P3EODDP3M7NMQD5VHXHB4TDVUCWXOB6JB4HCLSFS6HLA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.883386212 (493h22m3.754469247s) [noise=true , address=145.79.227.36:4517 , piece_id=W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.884106413 (493h22m3.755189458s) [noise=true , address=145.79.227.11:4730 , piece_id=CEK3H5R4ZEBZ4YEHSXUYE5YHZ5DNNZKIR3Z3V7EXQ4YKNJ3AKKZA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.908129688 (493h22m3.779212723s) [noise=true , address=145.79.227.38:4977 , piece_id=DAIFVBQ3USULKMUBMMMQ6QEM6C3FYWR26A5YUSBUQW2SRROCOJAQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.910050525 (493h22m3.781133571s) [noise=true , address=145.79.227.49:4339 , piece_id=L2F3BLLRK5J5XZZL2RXRINDPVDOF3LSAE3IELCY3VSFC54VN6GIA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.916079365 (493h22m3.78716241s) [noise=true , address=145.79.227.50:4374 , piece_id=65RLVWP4VY3P5DZX3XNFR34RQIGMKP2T4T7MD2Y6IFVUS3W2VKXQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.969444722 (493h22m3.840527767s) [noise=true , address=45.91.116.13:4154 , piece_id=LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.997975781 (493h22m3.869058826s) [noise=true , address=145.79.227.43:4157 , piece_id=37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.025734568 (493h22m3.896817613s) [noise=true , address=145.79.227.4:4185 , piece_id=EJ2CPERUCGKDIVIFA3JB3D46QLMZIUKYCLWNOVZ4WZ6QU5DPH4FA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.054893371 (493h22m3.925976406s) [noise=true , address=77.90.31.11:4556 , piece_id=EXB3WOSS4NUNSM2SPREQVSTD73RP22LY7CO462DXH5MYDAWF7WFA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.087543273 (493h22m3.958626318s) [noise=true , address=45.91.116.9:4263 , piece_id=JBCZMTOD6NRXPBN3SXZMR7IE27VAUNSB2IJK4X3RJ7YJFRWWVYQQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193060726 (493h22m4.064143761s) [noise=true , address=145.79.227.5:4223 , piece_id=32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193488190 (493h22m4.064571236s) [noise=true , address=145.79.227.6:4257 , piece_id=2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193515361 (493h22m4.064598406s) [seg_pos={0 0}, upload=433027]: Segment reached optimal threshold of 13 pieces. Dec 26 04:51:04.193755188 (493h22m4.064838233s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection Dec 26 04:51:04.193764265 (493h22m4.0648473s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.193942395 (493h22m4.06502544s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection Dec 26 04:51:04.193954127 (493h22m4.065037172s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.194259178 (493h22m4.065342223s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 26 04:51:04.194269337 (493h22m4.065352383s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.194527208 (493h22m4.065610253s) [seg_pos={0 0}, upload=433027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:51:04.194670963 (493h22m4.065754008s) [upload=433027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:51:04.194678186 (493h22m4.065761221s) [upload=433027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:51:05.236659258 (493h22m5.107742303s) [upload=433027]: Done uploading segments... Dec 26 04:51:05.236684847 (493h22m5.107767882s) [upload=433027]: Upload finished. err: <nil> Duration: 5.971966913s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 34.629904412s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:54:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 34.629938395s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.194318864s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:54:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.194364712s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.................X..X..X.X....XX.X.X.X.XX.X......X.....X....] duration: 17.436348291s trace: https://jaeger.dev.storj.io/trace/47b8400ca92cf8ee FAIL (2025-12-26 04:54:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, Piece ID: 2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: 37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: 32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ) context deadline exceeded ======LOGS====== h22m1.325682255s) [upload=433027]: Uploading segments... Dec 26 04:51:01.454784964 (493h22m1.325868009s) [upload=433027]: Got next segment. Inline: false Dec 26 04:51:01.454842182 (493h22m1.325925227s) [upload=433027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:51:01.454848625 (493h22m1.32593167s) [upload=433027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:51:01.716256741 (493h22m1.587339786s) [seg_pos={0 0}, upload=433027]: Begin upload segment... Dec 26 04:51:01.716321104 (493h22m1.587404149s) [seg_pos={0 0}, upload=433027]: Done begin upload segment. Dec 26 04:51:01.716350710 (493h22m1.587433755s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.716416826 (493h22m1.587499871s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.717137017 (493h22m1.588220062s) [noise=true , address=145.79.227.4:4185 , piece_id=EJ2CPERUCGKDIVIFA3JB3D46QLMZIUKYCLWNOVZ4WZ6QU5DPH4FA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.717275030 (493h22m1.588358075s) [noise=true , address=145.79.227.38:4977 , piece_id=DAIFVBQ3USULKMUBMMMQ6QEM6C3FYWR26A5YUSBUQW2SRROCOJAQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718006162 (493h22m1.589089207s) [noise=true , address=145.79.227.43:4157 , piece_id=37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718273662 (493h22m1.589356707s) [noise=true , address=45.91.116.13:4154 , piece_id=LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718632625 (493h22m1.58971566s) [noise=true , address=145.79.227.5:4223 , piece_id=32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.719228028 (493h22m1.590311074s) [noise=true , address=145.79.227.49:4339 , piece_id=L2F3BLLRK5J5XZZL2RXRINDPVDOF3LSAE3IELCY3VSFC54VN6GIA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.720480954 (493h22m1.591563999s) [noise=true , address=145.79.227.36:4517 , piece_id=W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.721203200 (493h22m1.592286245s) [noise=true , address=145.79.227.50:4374 , piece_id=65RLVWP4VY3P5DZX3XNFR34RQIGMKP2T4T7MD2Y6IFVUS3W2VKXQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.721530704 (493h22m1.592613739s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.722180451 (493h22m1.593263496s) [noise=true , address=45.91.116.9:4263 , piece_id=JBCZMTOD6NRXPBN3SXZMR7IE27VAUNSB2IJK4X3RJ7YJFRWWVYQQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.723072510 (493h22m1.594155545s) [noise=true , address=145.79.227.11:4730 , piece_id=CEK3H5R4ZEBZ4YEHSXUYE5YHZ5DNNZKIR3Z3V7EXQ4YKNJ3AKKZA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.723594634 (493h22m1.594677679s) [noise=true , address=145.79.227.51:4411 , piece_id=HANFRVC6P3EODDP3M7NMQD5VHXHB4TDVUCWXOB6JB4HCLSFS6HLA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.724105607 (493h22m1.595188652s) [noise=true , address=77.90.31.11:4556 , piece_id=EXB3WOSS4NUNSM2SPREQVSTD73RP22LY7CO462DXH5MYDAWF7WFA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.724578727 (493h22m1.595661772s) [noise=true , address=145.79.227.6:4257 , piece_id=2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:03.471291472 (493h22m3.342374517s) [upload=433027]: Next returned nil segment Dec 26 04:51:03.471307903 (493h22m3.342390938s) [upload=433027]: Waiting for error group managing segments... Dec 26 04:51:03.881101321 (493h22m3.752184366s) [noise=true , address=145.79.227.51:4411 , piece_id=HANFRVC6P3EODDP3M7NMQD5VHXHB4TDVUCWXOB6JB4HCLSFS6HLA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.883386212 (493h22m3.754469247s) [noise=true , address=145.79.227.36:4517 , piece_id=W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.884106413 (493h22m3.755189458s) [noise=true , address=145.79.227.11:4730 , piece_id=CEK3H5R4ZEBZ4YEHSXUYE5YHZ5DNNZKIR3Z3V7EXQ4YKNJ3AKKZA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.908129688 (493h22m3.779212723s) [noise=true , address=145.79.227.38:4977 , piece_id=DAIFVBQ3USULKMUBMMMQ6QEM6C3FYWR26A5YUSBUQW2SRROCOJAQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.910050525 (493h22m3.781133571s) [noise=true , address=145.79.227.49:4339 , piece_id=L2F3BLLRK5J5XZZL2RXRINDPVDOF3LSAE3IELCY3VSFC54VN6GIA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.916079365 (493h22m3.78716241s) [noise=true , address=145.79.227.50:4374 , piece_id=65RLVWP4VY3P5DZX3XNFR34RQIGMKP2T4T7MD2Y6IFVUS3W2VKXQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.969444722 (493h22m3.840527767s) [noise=true , address=45.91.116.13:4154 , piece_id=LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.997975781 (493h22m3.869058826s) [noise=true , address=145.79.227.43:4157 , piece_id=37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.025734568 (493h22m3.896817613s) [noise=true , address=145.79.227.4:4185 , piece_id=EJ2CPERUCGKDIVIFA3JB3D46QLMZIUKYCLWNOVZ4WZ6QU5DPH4FA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.054893371 (493h22m3.925976406s) [noise=true , address=77.90.31.11:4556 , piece_id=EXB3WOSS4NUNSM2SPREQVSTD73RP22LY7CO462DXH5MYDAWF7WFA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.087543273 (493h22m3.958626318s) [noise=true , address=45.91.116.9:4263 , piece_id=JBCZMTOD6NRXPBN3SXZMR7IE27VAUNSB2IJK4X3RJ7YJFRWWVYQQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193060726 (493h22m4.064143761s) [noise=true , address=145.79.227.5:4223 , piece_id=32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193488190 (493h22m4.064571236s) [noise=true , address=145.79.227.6:4257 , piece_id=2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193515361 (493h22m4.064598406s) [seg_pos={0 0}, upload=433027]: Segment reached optimal threshold of 13 pieces. Dec 26 04:51:04.193755188 (493h22m4.064838233s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection Dec 26 04:51:04.193764265 (493h22m4.0648473s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.193942395 (493h22m4.06502544s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection Dec 26 04:51:04.193954127 (493h22m4.065037172s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.194259178 (493h22m4.065342223s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 26 04:51:04.194269337 (493h22m4.065352383s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.194527208 (493h22m4.065610253s) [seg_pos={0 0}, upload=433027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:51:04.194670963 (493h22m4.065754008s) [upload=433027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:51:04.194678186 (493h22m4.065761221s) [upload=433027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:51:05.236659258 (493h22m5.107742303s) [upload=433027]: Done uploading segments... Dec 26 04:51:05.236684847 (493h22m5.107767882s) [upload=433027]: Upload finished. err: <nil> Duration: 17.436379451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 28.816685583s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:54:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 28.816706584s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.159257404s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:53:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.1592821s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..................X..X..X.X....XX.X.X.X.XX.X......X.....X...] duration: 22.012478172s trace: https://jaeger.dev.storj.io/trace/47b8400ca92cf8ee FAIL (2025-12-26 04:53:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, Piece ID: 2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: 37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: 32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ) context deadline exceeded ======LOGS====== h22m1.325682255s) [upload=433027]: Uploading segments... Dec 26 04:51:01.454784964 (493h22m1.325868009s) [upload=433027]: Got next segment. Inline: false Dec 26 04:51:01.454842182 (493h22m1.325925227s) [upload=433027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:51:01.454848625 (493h22m1.32593167s) [upload=433027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:51:01.716256741 (493h22m1.587339786s) [seg_pos={0 0}, upload=433027]: Begin upload segment... Dec 26 04:51:01.716321104 (493h22m1.587404149s) [seg_pos={0 0}, upload=433027]: Done begin upload segment. Dec 26 04:51:01.716350710 (493h22m1.587433755s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.716416826 (493h22m1.587499871s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.717137017 (493h22m1.588220062s) [noise=true , address=145.79.227.4:4185 , piece_id=EJ2CPERUCGKDIVIFA3JB3D46QLMZIUKYCLWNOVZ4WZ6QU5DPH4FA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.717275030 (493h22m1.588358075s) [noise=true , address=145.79.227.38:4977 , piece_id=DAIFVBQ3USULKMUBMMMQ6QEM6C3FYWR26A5YUSBUQW2SRROCOJAQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718006162 (493h22m1.589089207s) [noise=true , address=145.79.227.43:4157 , piece_id=37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718273662 (493h22m1.589356707s) [noise=true , address=45.91.116.13:4154 , piece_id=LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718632625 (493h22m1.58971566s) [noise=true , address=145.79.227.5:4223 , piece_id=32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.719228028 (493h22m1.590311074s) [noise=true , address=145.79.227.49:4339 , piece_id=L2F3BLLRK5J5XZZL2RXRINDPVDOF3LSAE3IELCY3VSFC54VN6GIA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.720480954 (493h22m1.591563999s) [noise=true , address=145.79.227.36:4517 , piece_id=W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.721203200 (493h22m1.592286245s) [noise=true , address=145.79.227.50:4374 , piece_id=65RLVWP4VY3P5DZX3XNFR34RQIGMKP2T4T7MD2Y6IFVUS3W2VKXQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.721530704 (493h22m1.592613739s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.722180451 (493h22m1.593263496s) [noise=true , address=45.91.116.9:4263 , piece_id=JBCZMTOD6NRXPBN3SXZMR7IE27VAUNSB2IJK4X3RJ7YJFRWWVYQQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.723072510 (493h22m1.594155545s) [noise=true , address=145.79.227.11:4730 , piece_id=CEK3H5R4ZEBZ4YEHSXUYE5YHZ5DNNZKIR3Z3V7EXQ4YKNJ3AKKZA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.723594634 (493h22m1.594677679s) [noise=true , address=145.79.227.51:4411 , piece_id=HANFRVC6P3EODDP3M7NMQD5VHXHB4TDVUCWXOB6JB4HCLSFS6HLA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.724105607 (493h22m1.595188652s) [noise=true , address=77.90.31.11:4556 , piece_id=EXB3WOSS4NUNSM2SPREQVSTD73RP22LY7CO462DXH5MYDAWF7WFA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.724578727 (493h22m1.595661772s) [noise=true , address=145.79.227.6:4257 , piece_id=2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:03.471291472 (493h22m3.342374517s) [upload=433027]: Next returned nil segment Dec 26 04:51:03.471307903 (493h22m3.342390938s) [upload=433027]: Waiting for error group managing segments... Dec 26 04:51:03.881101321 (493h22m3.752184366s) [noise=true , address=145.79.227.51:4411 , piece_id=HANFRVC6P3EODDP3M7NMQD5VHXHB4TDVUCWXOB6JB4HCLSFS6HLA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.883386212 (493h22m3.754469247s) [noise=true , address=145.79.227.36:4517 , piece_id=W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.884106413 (493h22m3.755189458s) [noise=true , address=145.79.227.11:4730 , piece_id=CEK3H5R4ZEBZ4YEHSXUYE5YHZ5DNNZKIR3Z3V7EXQ4YKNJ3AKKZA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.908129688 (493h22m3.779212723s) [noise=true , address=145.79.227.38:4977 , piece_id=DAIFVBQ3USULKMUBMMMQ6QEM6C3FYWR26A5YUSBUQW2SRROCOJAQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.910050525 (493h22m3.781133571s) [noise=true , address=145.79.227.49:4339 , piece_id=L2F3BLLRK5J5XZZL2RXRINDPVDOF3LSAE3IELCY3VSFC54VN6GIA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.916079365 (493h22m3.78716241s) [noise=true , address=145.79.227.50:4374 , piece_id=65RLVWP4VY3P5DZX3XNFR34RQIGMKP2T4T7MD2Y6IFVUS3W2VKXQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.969444722 (493h22m3.840527767s) [noise=true , address=45.91.116.13:4154 , piece_id=LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.997975781 (493h22m3.869058826s) [noise=true , address=145.79.227.43:4157 , piece_id=37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.025734568 (493h22m3.896817613s) [noise=true , address=145.79.227.4:4185 , piece_id=EJ2CPERUCGKDIVIFA3JB3D46QLMZIUKYCLWNOVZ4WZ6QU5DPH4FA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.054893371 (493h22m3.925976406s) [noise=true , address=77.90.31.11:4556 , piece_id=EXB3WOSS4NUNSM2SPREQVSTD73RP22LY7CO462DXH5MYDAWF7WFA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.087543273 (493h22m3.958626318s) [noise=true , address=45.91.116.9:4263 , piece_id=JBCZMTOD6NRXPBN3SXZMR7IE27VAUNSB2IJK4X3RJ7YJFRWWVYQQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193060726 (493h22m4.064143761s) [noise=true , address=145.79.227.5:4223 , piece_id=32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193488190 (493h22m4.064571236s) [noise=true , address=145.79.227.6:4257 , piece_id=2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193515361 (493h22m4.064598406s) [seg_pos={0 0}, upload=433027]: Segment reached optimal threshold of 13 pieces. Dec 26 04:51:04.193755188 (493h22m4.064838233s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection Dec 26 04:51:04.193764265 (493h22m4.0648473s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.193942395 (493h22m4.06502544s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection Dec 26 04:51:04.193954127 (493h22m4.065037172s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.194259178 (493h22m4.065342223s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 26 04:51:04.194269337 (493h22m4.065352383s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.194527208 (493h22m4.065610253s) [seg_pos={0 0}, upload=433027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:51:04.194670963 (493h22m4.065754008s) [upload=433027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:51:04.194678186 (493h22m4.065761221s) [upload=433027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:51:05.236659258 (493h22m5.107742303s) [upload=433027]: Done uploading segments... Dec 26 04:51:05.236684847 (493h22m5.107767882s) [upload=433027]: Upload finished. err: <nil> Duration: 22.012495285s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.321749626s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:53:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.321772089s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 15.366774595s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:53:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 15.366806345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.555071237s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:52:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.55508841s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 10.206953666s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:52:35 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 10.206982171s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 25.195935568s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:52:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 25.195957829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...................X..X..X.X....XX.X.X.X.XX.X......X.....X..] duration: 5.784453723s trace: https://jaeger.dev.storj.io/trace/47b8400ca92cf8ee FAIL (2025-12-26 04:52:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, Piece ID: 2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: 37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: 32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ) context deadline exceeded ======LOGS====== h22m1.325682255s) [upload=433027]: Uploading segments... Dec 26 04:51:01.454784964 (493h22m1.325868009s) [upload=433027]: Got next segment. Inline: false Dec 26 04:51:01.454842182 (493h22m1.325925227s) [upload=433027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:51:01.454848625 (493h22m1.32593167s) [upload=433027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:51:01.716256741 (493h22m1.587339786s) [seg_pos={0 0}, upload=433027]: Begin upload segment... Dec 26 04:51:01.716321104 (493h22m1.587404149s) [seg_pos={0 0}, upload=433027]: Done begin upload segment. Dec 26 04:51:01.716350710 (493h22m1.587433755s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.716416826 (493h22m1.587499871s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.717137017 (493h22m1.588220062s) [noise=true , address=145.79.227.4:4185 , piece_id=EJ2CPERUCGKDIVIFA3JB3D46QLMZIUKYCLWNOVZ4WZ6QU5DPH4FA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.717275030 (493h22m1.588358075s) [noise=true , address=145.79.227.38:4977 , piece_id=DAIFVBQ3USULKMUBMMMQ6QEM6C3FYWR26A5YUSBUQW2SRROCOJAQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718006162 (493h22m1.589089207s) [noise=true , address=145.79.227.43:4157 , piece_id=37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718273662 (493h22m1.589356707s) [noise=true , address=45.91.116.13:4154 , piece_id=LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718632625 (493h22m1.58971566s) [noise=true , address=145.79.227.5:4223 , piece_id=32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.719228028 (493h22m1.590311074s) [noise=true , address=145.79.227.49:4339 , piece_id=L2F3BLLRK5J5XZZL2RXRINDPVDOF3LSAE3IELCY3VSFC54VN6GIA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.720480954 (493h22m1.591563999s) [noise=true , address=145.79.227.36:4517 , piece_id=W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.721203200 (493h22m1.592286245s) [noise=true , address=145.79.227.50:4374 , piece_id=65RLVWP4VY3P5DZX3XNFR34RQIGMKP2T4T7MD2Y6IFVUS3W2VKXQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.721530704 (493h22m1.592613739s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.722180451 (493h22m1.593263496s) [noise=true , address=45.91.116.9:4263 , piece_id=JBCZMTOD6NRXPBN3SXZMR7IE27VAUNSB2IJK4X3RJ7YJFRWWVYQQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.723072510 (493h22m1.594155545s) [noise=true , address=145.79.227.11:4730 , piece_id=CEK3H5R4ZEBZ4YEHSXUYE5YHZ5DNNZKIR3Z3V7EXQ4YKNJ3AKKZA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.723594634 (493h22m1.594677679s) [noise=true , address=145.79.227.51:4411 , piece_id=HANFRVC6P3EODDP3M7NMQD5VHXHB4TDVUCWXOB6JB4HCLSFS6HLA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.724105607 (493h22m1.595188652s) [noise=true , address=77.90.31.11:4556 , piece_id=EXB3WOSS4NUNSM2SPREQVSTD73RP22LY7CO462DXH5MYDAWF7WFA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.724578727 (493h22m1.595661772s) [noise=true , address=145.79.227.6:4257 , piece_id=2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:03.471291472 (493h22m3.342374517s) [upload=433027]: Next returned nil segment Dec 26 04:51:03.471307903 (493h22m3.342390938s) [upload=433027]: Waiting for error group managing segments... Dec 26 04:51:03.881101321 (493h22m3.752184366s) [noise=true , address=145.79.227.51:4411 , piece_id=HANFRVC6P3EODDP3M7NMQD5VHXHB4TDVUCWXOB6JB4HCLSFS6HLA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.883386212 (493h22m3.754469247s) [noise=true , address=145.79.227.36:4517 , piece_id=W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.884106413 (493h22m3.755189458s) [noise=true , address=145.79.227.11:4730 , piece_id=CEK3H5R4ZEBZ4YEHSXUYE5YHZ5DNNZKIR3Z3V7EXQ4YKNJ3AKKZA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.908129688 (493h22m3.779212723s) [noise=true , address=145.79.227.38:4977 , piece_id=DAIFVBQ3USULKMUBMMMQ6QEM6C3FYWR26A5YUSBUQW2SRROCOJAQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.910050525 (493h22m3.781133571s) [noise=true , address=145.79.227.49:4339 , piece_id=L2F3BLLRK5J5XZZL2RXRINDPVDOF3LSAE3IELCY3VSFC54VN6GIA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.916079365 (493h22m3.78716241s) [noise=true , address=145.79.227.50:4374 , piece_id=65RLVWP4VY3P5DZX3XNFR34RQIGMKP2T4T7MD2Y6IFVUS3W2VKXQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.969444722 (493h22m3.840527767s) [noise=true , address=45.91.116.13:4154 , piece_id=LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.997975781 (493h22m3.869058826s) [noise=true , address=145.79.227.43:4157 , piece_id=37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.025734568 (493h22m3.896817613s) [noise=true , address=145.79.227.4:4185 , piece_id=EJ2CPERUCGKDIVIFA3JB3D46QLMZIUKYCLWNOVZ4WZ6QU5DPH4FA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.054893371 (493h22m3.925976406s) [noise=true , address=77.90.31.11:4556 , piece_id=EXB3WOSS4NUNSM2SPREQVSTD73RP22LY7CO462DXH5MYDAWF7WFA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.087543273 (493h22m3.958626318s) [noise=true , address=45.91.116.9:4263 , piece_id=JBCZMTOD6NRXPBN3SXZMR7IE27VAUNSB2IJK4X3RJ7YJFRWWVYQQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193060726 (493h22m4.064143761s) [noise=true , address=145.79.227.5:4223 , piece_id=32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193488190 (493h22m4.064571236s) [noise=true , address=145.79.227.6:4257 , piece_id=2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193515361 (493h22m4.064598406s) [seg_pos={0 0}, upload=433027]: Segment reached optimal threshold of 13 pieces. Dec 26 04:51:04.193755188 (493h22m4.064838233s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection Dec 26 04:51:04.193764265 (493h22m4.0648473s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.193942395 (493h22m4.06502544s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection Dec 26 04:51:04.193954127 (493h22m4.065037172s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.194259178 (493h22m4.065342223s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 26 04:51:04.194269337 (493h22m4.065352383s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.194527208 (493h22m4.065610253s) [seg_pos={0 0}, upload=433027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:51:04.194670963 (493h22m4.065754008s) [upload=433027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:51:04.194678186 (493h22m4.065761221s) [upload=433027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:51:05.236659258 (493h22m5.107742303s) [upload=433027]: Done uploading segments... Dec 26 04:51:05.236684847 (493h22m5.107767882s) [upload=433027]: Upload finished. err: <nil> Duration: 5.784480945s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.607270805s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:52:09 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.607327044s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....................X..X..X.X....XX.X.X.X.XX.X......X.....X.] duration: 11.851536438s trace: https://jaeger.dev.storj.io/trace/47b8400ca92cf8ee FAIL (2025-12-26 04:51:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, Piece ID: 2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: 37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: 32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ) context deadline exceeded ======LOGS====== h22m1.325682255s) [upload=433027]: Uploading segments... Dec 26 04:51:01.454784964 (493h22m1.325868009s) [upload=433027]: Got next segment. Inline: false Dec 26 04:51:01.454842182 (493h22m1.325925227s) [upload=433027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:51:01.454848625 (493h22m1.32593167s) [upload=433027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:51:01.716256741 (493h22m1.587339786s) [seg_pos={0 0}, upload=433027]: Begin upload segment... Dec 26 04:51:01.716321104 (493h22m1.587404149s) [seg_pos={0 0}, upload=433027]: Done begin upload segment. Dec 26 04:51:01.716350710 (493h22m1.587433755s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.716416826 (493h22m1.587499871s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.717137017 (493h22m1.588220062s) [noise=true , address=145.79.227.4:4185 , piece_id=EJ2CPERUCGKDIVIFA3JB3D46QLMZIUKYCLWNOVZ4WZ6QU5DPH4FA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.717275030 (493h22m1.588358075s) [noise=true , address=145.79.227.38:4977 , piece_id=DAIFVBQ3USULKMUBMMMQ6QEM6C3FYWR26A5YUSBUQW2SRROCOJAQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718006162 (493h22m1.589089207s) [noise=true , address=145.79.227.43:4157 , piece_id=37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718273662 (493h22m1.589356707s) [noise=true , address=45.91.116.13:4154 , piece_id=LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718632625 (493h22m1.58971566s) [noise=true , address=145.79.227.5:4223 , piece_id=32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.719228028 (493h22m1.590311074s) [noise=true , address=145.79.227.49:4339 , piece_id=L2F3BLLRK5J5XZZL2RXRINDPVDOF3LSAE3IELCY3VSFC54VN6GIA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.720480954 (493h22m1.591563999s) [noise=true , address=145.79.227.36:4517 , piece_id=W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.721203200 (493h22m1.592286245s) [noise=true , address=145.79.227.50:4374 , piece_id=65RLVWP4VY3P5DZX3XNFR34RQIGMKP2T4T7MD2Y6IFVUS3W2VKXQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.721530704 (493h22m1.592613739s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.722180451 (493h22m1.593263496s) [noise=true , address=45.91.116.9:4263 , piece_id=JBCZMTOD6NRXPBN3SXZMR7IE27VAUNSB2IJK4X3RJ7YJFRWWVYQQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.723072510 (493h22m1.594155545s) [noise=true , address=145.79.227.11:4730 , piece_id=CEK3H5R4ZEBZ4YEHSXUYE5YHZ5DNNZKIR3Z3V7EXQ4YKNJ3AKKZA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.723594634 (493h22m1.594677679s) [noise=true , address=145.79.227.51:4411 , piece_id=HANFRVC6P3EODDP3M7NMQD5VHXHB4TDVUCWXOB6JB4HCLSFS6HLA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.724105607 (493h22m1.595188652s) [noise=true , address=77.90.31.11:4556 , piece_id=EXB3WOSS4NUNSM2SPREQVSTD73RP22LY7CO462DXH5MYDAWF7WFA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.724578727 (493h22m1.595661772s) [noise=true , address=145.79.227.6:4257 , piece_id=2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:03.471291472 (493h22m3.342374517s) [upload=433027]: Next returned nil segment Dec 26 04:51:03.471307903 (493h22m3.342390938s) [upload=433027]: Waiting for error group managing segments... Dec 26 04:51:03.881101321 (493h22m3.752184366s) [noise=true , address=145.79.227.51:4411 , piece_id=HANFRVC6P3EODDP3M7NMQD5VHXHB4TDVUCWXOB6JB4HCLSFS6HLA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.883386212 (493h22m3.754469247s) [noise=true , address=145.79.227.36:4517 , piece_id=W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.884106413 (493h22m3.755189458s) [noise=true , address=145.79.227.11:4730 , piece_id=CEK3H5R4ZEBZ4YEHSXUYE5YHZ5DNNZKIR3Z3V7EXQ4YKNJ3AKKZA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.908129688 (493h22m3.779212723s) [noise=true , address=145.79.227.38:4977 , piece_id=DAIFVBQ3USULKMUBMMMQ6QEM6C3FYWR26A5YUSBUQW2SRROCOJAQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.910050525 (493h22m3.781133571s) [noise=true , address=145.79.227.49:4339 , piece_id=L2F3BLLRK5J5XZZL2RXRINDPVDOF3LSAE3IELCY3VSFC54VN6GIA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.916079365 (493h22m3.78716241s) [noise=true , address=145.79.227.50:4374 , piece_id=65RLVWP4VY3P5DZX3XNFR34RQIGMKP2T4T7MD2Y6IFVUS3W2VKXQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.969444722 (493h22m3.840527767s) [noise=true , address=45.91.116.13:4154 , piece_id=LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.997975781 (493h22m3.869058826s) [noise=true , address=145.79.227.43:4157 , piece_id=37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.025734568 (493h22m3.896817613s) [noise=true , address=145.79.227.4:4185 , piece_id=EJ2CPERUCGKDIVIFA3JB3D46QLMZIUKYCLWNOVZ4WZ6QU5DPH4FA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.054893371 (493h22m3.925976406s) [noise=true , address=77.90.31.11:4556 , piece_id=EXB3WOSS4NUNSM2SPREQVSTD73RP22LY7CO462DXH5MYDAWF7WFA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.087543273 (493h22m3.958626318s) [noise=true , address=45.91.116.9:4263 , piece_id=JBCZMTOD6NRXPBN3SXZMR7IE27VAUNSB2IJK4X3RJ7YJFRWWVYQQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193060726 (493h22m4.064143761s) [noise=true , address=145.79.227.5:4223 , piece_id=32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193488190 (493h22m4.064571236s) [noise=true , address=145.79.227.6:4257 , piece_id=2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193515361 (493h22m4.064598406s) [seg_pos={0 0}, upload=433027]: Segment reached optimal threshold of 13 pieces. Dec 26 04:51:04.193755188 (493h22m4.064838233s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection Dec 26 04:51:04.193764265 (493h22m4.0648473s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.193942395 (493h22m4.06502544s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection Dec 26 04:51:04.193954127 (493h22m4.065037172s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.194259178 (493h22m4.065342223s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 26 04:51:04.194269337 (493h22m4.065352383s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.194527208 (493h22m4.065610253s) [seg_pos={0 0}, upload=433027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:51:04.194670963 (493h22m4.065754008s) [upload=433027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:51:04.194678186 (493h22m4.065761221s) [upload=433027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:51:05.236659258 (493h22m5.107742303s) [upload=433027]: Done uploading segments... Dec 26 04:51:05.236684847 (493h22m5.107767882s) [upload=433027]: Upload finished. err: <nil> Duration: 11.851562097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 33.477003148s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:51:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 33.477034279s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.267759064s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:51:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.267784212s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....................X..X..X.X....XX.X.X.X.XX.X......X.....X] duration: 30.002833546s trace: https://jaeger.dev.storj.io/trace/47b8400ca92cf8ee FAIL (2025-12-26 04:51:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, Piece ID: 2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: 37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: 32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ) context deadline exceeded ======LOGS====== h22m1.325682255s) [upload=433027]: Uploading segments... Dec 26 04:51:01.454784964 (493h22m1.325868009s) [upload=433027]: Got next segment. Inline: false Dec 26 04:51:01.454842182 (493h22m1.325925227s) [upload=433027]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:51:01.454848625 (493h22m1.32593167s) [upload=433027]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:51:01.716256741 (493h22m1.587339786s) [seg_pos={0 0}, upload=433027]: Begin upload segment... Dec 26 04:51:01.716321104 (493h22m1.587404149s) [seg_pos={0 0}, upload=433027]: Done begin upload segment. Dec 26 04:51:01.716350710 (493h22m1.587433755s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.716416826 (493h22m1.587499871s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.717137017 (493h22m1.588220062s) [noise=true , address=145.79.227.4:4185 , piece_id=EJ2CPERUCGKDIVIFA3JB3D46QLMZIUKYCLWNOVZ4WZ6QU5DPH4FA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.717275030 (493h22m1.588358075s) [noise=true , address=145.79.227.38:4977 , piece_id=DAIFVBQ3USULKMUBMMMQ6QEM6C3FYWR26A5YUSBUQW2SRROCOJAQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718006162 (493h22m1.589089207s) [noise=true , address=145.79.227.43:4157 , piece_id=37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718273662 (493h22m1.589356707s) [noise=true , address=45.91.116.13:4154 , piece_id=LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.718632625 (493h22m1.58971566s) [noise=true , address=145.79.227.5:4223 , piece_id=32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.719228028 (493h22m1.590311074s) [noise=true , address=145.79.227.49:4339 , piece_id=L2F3BLLRK5J5XZZL2RXRINDPVDOF3LSAE3IELCY3VSFC54VN6GIA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.720480954 (493h22m1.591563999s) [noise=true , address=145.79.227.36:4517 , piece_id=W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.721203200 (493h22m1.592286245s) [noise=true , address=145.79.227.50:4374 , piece_id=65RLVWP4VY3P5DZX3XNFR34RQIGMKP2T4T7MD2Y6IFVUS3W2VKXQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.721530704 (493h22m1.592613739s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.722180451 (493h22m1.593263496s) [noise=true , address=45.91.116.9:4263 , piece_id=JBCZMTOD6NRXPBN3SXZMR7IE27VAUNSB2IJK4X3RJ7YJFRWWVYQQ, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.723072510 (493h22m1.594155545s) [noise=true , address=145.79.227.11:4730 , piece_id=CEK3H5R4ZEBZ4YEHSXUYE5YHZ5DNNZKIR3Z3V7EXQ4YKNJ3AKKZA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.723594634 (493h22m1.594677679s) [noise=true , address=145.79.227.51:4411 , piece_id=HANFRVC6P3EODDP3M7NMQD5VHXHB4TDVUCWXOB6JB4HCLSFS6HLA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.724105607 (493h22m1.595188652s) [noise=true , address=77.90.31.11:4556 , piece_id=EXB3WOSS4NUNSM2SPREQVSTD73RP22LY7CO462DXH5MYDAWF7WFA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:01.724578727 (493h22m1.595661772s) [noise=true , address=145.79.227.6:4257 , piece_id=2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA, seg_pos={0 0}, upload=433027]: Uploading piece... Dec 26 04:51:03.471291472 (493h22m3.342374517s) [upload=433027]: Next returned nil segment Dec 26 04:51:03.471307903 (493h22m3.342390938s) [upload=433027]: Waiting for error group managing segments... Dec 26 04:51:03.881101321 (493h22m3.752184366s) [noise=true , address=145.79.227.51:4411 , piece_id=HANFRVC6P3EODDP3M7NMQD5VHXHB4TDVUCWXOB6JB4HCLSFS6HLA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.883386212 (493h22m3.754469247s) [noise=true , address=145.79.227.36:4517 , piece_id=W7GWO2BFGRIKJLCSAT3LHSEKQ7ET7IACBJVMQ3NN2NIUSOV7APXQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.884106413 (493h22m3.755189458s) [noise=true , address=145.79.227.11:4730 , piece_id=CEK3H5R4ZEBZ4YEHSXUYE5YHZ5DNNZKIR3Z3V7EXQ4YKNJ3AKKZA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.908129688 (493h22m3.779212723s) [noise=true , address=145.79.227.38:4977 , piece_id=DAIFVBQ3USULKMUBMMMQ6QEM6C3FYWR26A5YUSBUQW2SRROCOJAQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.910050525 (493h22m3.781133571s) [noise=true , address=145.79.227.49:4339 , piece_id=L2F3BLLRK5J5XZZL2RXRINDPVDOF3LSAE3IELCY3VSFC54VN6GIA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.916079365 (493h22m3.78716241s) [noise=true , address=145.79.227.50:4374 , piece_id=65RLVWP4VY3P5DZX3XNFR34RQIGMKP2T4T7MD2Y6IFVUS3W2VKXQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.969444722 (493h22m3.840527767s) [noise=true , address=45.91.116.13:4154 , piece_id=LYF73XN42DEMIWPDDHHFEUET4YMGCFQJA57EDJZWT6HQ4JPYRXTQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:03.997975781 (493h22m3.869058826s) [noise=true , address=145.79.227.43:4157 , piece_id=37YAJPL2J2TEQ2TUHNLRS246CMAOX7BXKGALSQGDFO2MRUTFOVVQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.025734568 (493h22m3.896817613s) [noise=true , address=145.79.227.4:4185 , piece_id=EJ2CPERUCGKDIVIFA3JB3D46QLMZIUKYCLWNOVZ4WZ6QU5DPH4FA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.054893371 (493h22m3.925976406s) [noise=true , address=77.90.31.11:4556 , piece_id=EXB3WOSS4NUNSM2SPREQVSTD73RP22LY7CO462DXH5MYDAWF7WFA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.087543273 (493h22m3.958626318s) [noise=true , address=45.91.116.9:4263 , piece_id=JBCZMTOD6NRXPBN3SXZMR7IE27VAUNSB2IJK4X3RJ7YJFRWWVYQQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193060726 (493h22m4.064143761s) [noise=true , address=145.79.227.5:4223 , piece_id=32AIFZME4VRNPQU2JKXAL3LQND4DT5U3EKC6TUX6S7DF7ZQ27KVA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193488190 (493h22m4.064571236s) [noise=true , address=145.79.227.6:4257 , piece_id=2ILB7HDWF35ULYM5TA5X7267YGFKQHCIL6XAAM3FJED4FG37Z7NA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: <nil> Dec 26 04:51:04.193515361 (493h22m4.064598406s) [seg_pos={0 0}, upload=433027]: Segment reached optimal threshold of 13 pieces. Dec 26 04:51:04.193755188 (493h22m4.064838233s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54506->77.90.31.17:4773: use of closed network connection Dec 26 04:51:04.193764265 (493h22m4.0648473s) [noise=true , address=77.90.31.17:4773 , piece_id=DJKL75BZZONZMQPRMC5BMZBLED3Z2N7UK4JHX5MN4FEQYFOFRB2Q, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.193942395 (493h22m4.06502544s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49740->145.79.227.10:4402: use of closed network connection Dec 26 04:51:04.193954127 (493h22m4.065037172s) [noise=true , address=145.79.227.10:4402 , piece_id=T2EXNYIQJNITQAD4GSBGWVC3YDGHJ55LOFWIAMOWHTTQQ6DLSNHA, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.194259178 (493h22m4.065342223s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 26 04:51:04.194269337 (493h22m4.065352383s) [noise=true , address=145.79.227.15:4876 , piece_id=JBUKOCSAADLAWNC76KSTL53ZATSADSQ4YBZVE6KBYUTG3N7JJAAQ, seg_pos={0 0}, upload=433027]: Long tail cancellation due to optimal threshold Dec 26 04:51:04.194527208 (493h22m4.065610253s) [seg_pos={0 0}, upload=433027]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:51:04.194670963 (493h22m4.065754008s) [upload=433027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:51:04.194678186 (493h22m4.065761221s) [upload=433027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:51:05.236659258 (493h22m5.107742303s) [upload=433027]: Done uploading segments... Dec 26 04:51:05.236684847 (493h22m5.107767882s) [upload=433027]: Upload finished. err: <nil> Duration: 30.002855318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 28.931801103s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:50:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 28.931838144s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.196993453s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:50:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.19702382s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......................X..X..X.X....XX.X.X.X.XX.X......X.....] duration: 6.120850513s trace: https://jaeger.dev.storj.io/trace/7fd1f6e029d2997a FAIL (2025-12-26 04:50:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, Piece ID: 33PLGGV5LFV2MQ5NIAGFDW3NZEDBUOPREN2EGNVKUYPNM76E4RAQ) context deadline exceeded; piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: 4OC43GK3EPEXGNVDBVCLUTQWWKVOTYP3YXT3P3EKCEL5FTYQNRSA) context deadline exceeded; piecestore: (Node ID: 1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d, Piece ID: RFEQYLZ5IKPEE6G2URQQZLBTLVYIJLZLREUISCPYVJXNXFVSZOHA) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: VBA5Z34H3ENLKKF5U3BUFXRN3TODH3N6EA2GPXVETZSQQN53PYGQ) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: QNQAWWDM2FOPRQ45KCY466RB5WW3CZMS2OADEXADNWYCL3HGE2DA) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: O5P7VKX3ILX2OMI4U3CYS2CDGWFILNQ4S3IMGU45MI4LNEEOLVMA) context deadline exceeded ======LOGS====== Dec 26 04:46:11.403823619 (493h17m11.274906665s) [upload=432954]: Starting upload Dec 26 04:46:11.403884125 (493h17m11.27496717s) [upload=432954]: Done starting upload Dec 26 04:46:11.404071982 (493h17m11.275155017s) [upload=432954]: Uploading segments... Dec 26 04:46:11.404120314 (493h17m11.275203349s) [upload=432954]: Got next segment. Inline: false Dec 26 04:46:11.404146584 (493h17m11.275229629s) [upload=432954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:46:11.404151002 (493h17m11.275234048s) [upload=432954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:46:11.643720875 (493h17m11.51480391s) [seg_pos={0 0}, upload=432954]: Begin upload segment... Dec 26 04:46:11.643771943 (493h17m11.514854978s) [seg_pos={0 0}, upload=432954]: Done begin upload segment. Dec 26 04:46:11.643815044 (493h17m11.514898079s) [noise=true , address=145.79.227.45:4230 , piece_id=QCJQZ4FYVPORIDUACXOUM6BNY55AAKLI7RA7XAPG3G5DDSAXZNNA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.643864538 (493h17m11.514947583s) [noise=true , address=145.79.227.35:4483 , piece_id=QNQAWWDM2FOPRQ45KCY466RB5WW3CZMS2OADEXADNWYCL3HGE2DA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.644634676 (493h17m11.515717711s) [noise=true , address=145.79.227.49:4338 , piece_id=O5P7VKX3ILX2OMI4U3CYS2CDGWFILNQ4S3IMGU45MI4LNEEOLVMA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.644880443 (493h17m11.515963489s) [noise=true , address=145.79.227.37:4566 , piece_id=QUYPK2QBYVUT6FCVJ3QNHJQZYW7KKJFH7TRB7MWGG5HFKQOFGVYA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.645033354 (493h17m11.516116389s) [noise=true , address=145.79.227.11:4732 , piece_id=BYOYWN56YERGV4T72GZTAZAGZ4PKNVO76AINW5APTS37NNWQWLIA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.645379403 (493h17m11.516462438s) [noise=true , address=145.79.227.5:4221 , piece_id=33PLGGV5LFV2MQ5NIAGFDW3NZEDBUOPREN2EGNVKUYPNM76E4RAQ, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.645466308 (493h17m11.516549343s) [noise=true , address=45.91.116.5:4299 , piece_id=EWLF23X32SDKY6KVSU4PUCZEYXWBJQW2OZCA57SACRQSLMUUYWKQ, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.645786348 (493h17m11.516869383s) [noise=true , address=145.79.227.4:4185 , piece_id=4OC43GK3EPEXGNVDBVCLUTQWWKVOTYP3YXT3P3EKCEL5FTYQNRSA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.646095778 (493h17m11.517178813s) [noise=true , address=145.79.227.10:4402 , piece_id=WLSY7UMISW4KS74K6QX5WL3K5FPX3APLEZXZYWFKCAR5CTPETW3Q, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.646499165 (493h17m11.51758221s) [noise=true , address=145.79.227.12:4767 , piece_id=BAMCB3OPKUOFWJOLCZKLOSGS4V5ZU6VXY6YI37UCDQBMSIKCTEBQ, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.646905189 (493h17m11.517988224s) [noise=true , address=145.79.227.46:4602 , piece_id=RFEQYLZ5IKPEE6G2URQQZLBTLVYIJLZLREUISCPYVJXNXFVSZOHA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.647052429 (493h17m11.518135474s) [noise=true , address=145.79.227.17:4970 , piece_id=VBA5Z34H3ENLKKF5U3BUFXRN3TODH3N6EA2GPXVETZSQQN53PYGQ, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.647302345 (493h17m11.51838539s) [noise=true , address=145.79.227.41:4085 , piece_id=CCQUJOZZXLLTKJJFXSIL7ERDJIIPXLF4D5UV7D67O7YRZFPZMV2A, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.647612465 (493h17m11.51869551s) [noise=true , address=145.79.227.16:4910 , piece_id=LSTHPP5EPYRTS6OLJ63S7JQ2LA2NUEIEWJGUY4C3LF3LTZHUCD7Q, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.648183694 (493h17m11.519266739s) [noise=true , address=145.79.227.47:4266 , piece_id=XWN34ED3GUVXA3PPTX5QU4KHR7CBBC745V44S3W57FBGRQ5I4HJA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.649790443 (493h17m11.520873488s) [noise=true , address=45.91.116.13:4153 , piece_id=ZK6U7BL3NYLN6Z5ZLS5BGUX5WEVFRSMVCN7ZRE35TOLTNTH46FDA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:13.268673497 (493h17m13.139756543s) [upload=432954]: Next returned nil segment Dec 26 04:46:13.268688636 (493h17m13.139771681s) [upload=432954]: Waiting for error group managing segments... Dec 26 04:46:13.647888619 (493h17m13.518971664s) [noise=true , address=145.79.227.12:4767 , piece_id=BAMCB3OPKUOFWJOLCZKLOSGS4V5ZU6VXY6YI37UCDQBMSIKCTEBQ, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.650591886 (493h17m13.521674921s) [noise=true , address=145.79.227.35:4483 , piece_id=QNQAWWDM2FOPRQ45KCY466RB5WW3CZMS2OADEXADNWYCL3HGE2DA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.651254949 (493h17m13.522337994s) [noise=true , address=145.79.227.45:4230 , piece_id=QCJQZ4FYVPORIDUACXOUM6BNY55AAKLI7RA7XAPG3G5DDSAXZNNA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.651861003 (493h17m13.522944038s) [noise=true , address=145.79.227.49:4338 , piece_id=O5P7VKX3ILX2OMI4U3CYS2CDGWFILNQ4S3IMGU45MI4LNEEOLVMA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.653880118 (493h17m13.524963153s) [noise=true , address=145.79.227.41:4085 , piece_id=CCQUJOZZXLLTKJJFXSIL7ERDJIIPXLF4D5UV7D67O7YRZFPZMV2A, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.654533863 (493h17m13.525616898s) [noise=true , address=145.79.227.46:4602 , piece_id=RFEQYLZ5IKPEE6G2URQQZLBTLVYIJLZLREUISCPYVJXNXFVSZOHA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.655418407 (493h17m13.526501442s) [noise=true , address=145.79.227.37:4566 , piece_id=QUYPK2QBYVUT6FCVJ3QNHJQZYW7KKJFH7TRB7MWGG5HFKQOFGVYA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.657324486 (493h17m13.528407521s) [noise=true , address=145.79.227.10:4402 , piece_id=WLSY7UMISW4KS74K6QX5WL3K5FPX3APLEZXZYWFKCAR5CTPETW3Q, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.660265606 (493h17m13.531348651s) [noise=true , address=145.79.227.11:4732 , piece_id=BYOYWN56YERGV4T72GZTAZAGZ4PKNVO76AINW5APTS37NNWQWLIA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.663478735 (493h17m13.53456177s) [noise=true , address=145.79.227.17:4970 , piece_id=VBA5Z34H3ENLKKF5U3BUFXRN3TODH3N6EA2GPXVETZSQQN53PYGQ, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.670422557 (493h17m13.541505602s) [noise=true , address=145.79.227.47:4266 , piece_id=XWN34ED3GUVXA3PPTX5QU4KHR7CBBC745V44S3W57FBGRQ5I4HJA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.674293307 (493h17m13.545376342s) [noise=true , address=145.79.227.5:4221 , piece_id=33PLGGV5LFV2MQ5NIAGFDW3NZEDBUOPREN2EGNVKUYPNM76E4RAQ, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.675494666 (493h17m13.546577711s) [noise=true , address=145.79.227.4:4185 , piece_id=4OC43GK3EPEXGNVDBVCLUTQWWKVOTYP3YXT3P3EKCEL5FTYQNRSA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.675504344 (493h17m13.546587379s) [seg_pos={0 0}, upload=432954]: Segment reached optimal threshold of 13 pieces. Dec 26 04:46:13.675660802 (493h17m13.546743847s) [noise=true , address=45.91.116.5:4299 , piece_id=EWLF23X32SDKY6KVSU4PUCZEYXWBJQW2OZCA57SACRQSLMUUYWKQ, seg_pos={0 0}, upload=432954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 04:46:13.675667525 (493h17m13.54675056s) [noise=true , address=45.91.116.5:4299 , piece_id=EWLF23X32SDKY6KVSU4PUCZEYXWBJQW2OZCA57SACRQSLMUUYWKQ, seg_pos={0 0}, upload=432954]: Long tail cancellation due to optimal threshold Dec 26 04:46:13.675867456 (493h17m13.546950501s) [noise=true , address=145.79.227.16:4910 , piece_id=LSTHPP5EPYRTS6OLJ63S7JQ2LA2NUEIEWJGUY4C3LF3LTZHUCD7Q, seg_pos={0 0}, upload=432954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 26 04:46:13.675874379 (493h17m13.546957424s) [noise=true , address=145.79.227.16:4910 , piece_id=LSTHPP5EPYRTS6OLJ63S7JQ2LA2NUEIEWJGUY4C3LF3LTZHUCD7Q, seg_pos={0 0}, upload=432954]: Long tail cancellation due to optimal threshold Dec 26 04:46:13.675960001 (493h17m13.547043036s) [noise=true , address=45.91.116.13:4153 , piece_id=ZK6U7BL3NYLN6Z5ZLS5BGUX5WEVFRSMVCN7ZRE35TOLTNTH46FDA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 04:46:13.675965011 (493h17m13.547048056s) [noise=true , address=45.91.116.13:4153 , piece_id=ZK6U7BL3NYLN6Z5ZLS5BGUX5WEVFRSMVCN7ZRE35TOLTNTH46FDA, seg_pos={0 0}, upload=432954]: Long tail cancellation due to optimal threshold Dec 26 04:46:13.675980591 (493h17m13.547063636s) [seg_pos={0 0}, upload=432954]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:46:13.676048029 (493h17m13.547131064s) [upload=432954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:46:13.676053339 (493h17m13.547136384s) [upload=432954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:46:13.955881701 (493h17m13.826964736s) [upload=432954]: Done uploading segments... Dec 26 04:46:13.955908562 (493h17m13.826991617s) [upload=432954]: Upload finished. err: <nil> Duration: 6.120878185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 33.514808726s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:49:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 33.514838002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......................X..X..X.X....XX.X.X.X.XX.X......X....] duration: 6.031074002s trace: https://jaeger.dev.storj.io/trace/7fd1f6e029d2997a FAIL (2025-12-26 04:49:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, Piece ID: 33PLGGV5LFV2MQ5NIAGFDW3NZEDBUOPREN2EGNVKUYPNM76E4RAQ) context deadline exceeded; piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: 4OC43GK3EPEXGNVDBVCLUTQWWKVOTYP3YXT3P3EKCEL5FTYQNRSA) context deadline exceeded; piecestore: (Node ID: 1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d, Piece ID: RFEQYLZ5IKPEE6G2URQQZLBTLVYIJLZLREUISCPYVJXNXFVSZOHA) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: VBA5Z34H3ENLKKF5U3BUFXRN3TODH3N6EA2GPXVETZSQQN53PYGQ) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: QNQAWWDM2FOPRQ45KCY466RB5WW3CZMS2OADEXADNWYCL3HGE2DA) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: O5P7VKX3ILX2OMI4U3CYS2CDGWFILNQ4S3IMGU45MI4LNEEOLVMA) context deadline exceeded ======LOGS====== Dec 26 04:46:11.403823619 (493h17m11.274906665s) [upload=432954]: Starting upload Dec 26 04:46:11.403884125 (493h17m11.27496717s) [upload=432954]: Done starting upload Dec 26 04:46:11.404071982 (493h17m11.275155017s) [upload=432954]: Uploading segments... Dec 26 04:46:11.404120314 (493h17m11.275203349s) [upload=432954]: Got next segment. Inline: false Dec 26 04:46:11.404146584 (493h17m11.275229629s) [upload=432954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:46:11.404151002 (493h17m11.275234048s) [upload=432954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:46:11.643720875 (493h17m11.51480391s) [seg_pos={0 0}, upload=432954]: Begin upload segment... Dec 26 04:46:11.643771943 (493h17m11.514854978s) [seg_pos={0 0}, upload=432954]: Done begin upload segment. Dec 26 04:46:11.643815044 (493h17m11.514898079s) [noise=true , address=145.79.227.45:4230 , piece_id=QCJQZ4FYVPORIDUACXOUM6BNY55AAKLI7RA7XAPG3G5DDSAXZNNA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.643864538 (493h17m11.514947583s) [noise=true , address=145.79.227.35:4483 , piece_id=QNQAWWDM2FOPRQ45KCY466RB5WW3CZMS2OADEXADNWYCL3HGE2DA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.644634676 (493h17m11.515717711s) [noise=true , address=145.79.227.49:4338 , piece_id=O5P7VKX3ILX2OMI4U3CYS2CDGWFILNQ4S3IMGU45MI4LNEEOLVMA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.644880443 (493h17m11.515963489s) [noise=true , address=145.79.227.37:4566 , piece_id=QUYPK2QBYVUT6FCVJ3QNHJQZYW7KKJFH7TRB7MWGG5HFKQOFGVYA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.645033354 (493h17m11.516116389s) [noise=true , address=145.79.227.11:4732 , piece_id=BYOYWN56YERGV4T72GZTAZAGZ4PKNVO76AINW5APTS37NNWQWLIA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.645379403 (493h17m11.516462438s) [noise=true , address=145.79.227.5:4221 , piece_id=33PLGGV5LFV2MQ5NIAGFDW3NZEDBUOPREN2EGNVKUYPNM76E4RAQ, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.645466308 (493h17m11.516549343s) [noise=true , address=45.91.116.5:4299 , piece_id=EWLF23X32SDKY6KVSU4PUCZEYXWBJQW2OZCA57SACRQSLMUUYWKQ, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.645786348 (493h17m11.516869383s) [noise=true , address=145.79.227.4:4185 , piece_id=4OC43GK3EPEXGNVDBVCLUTQWWKVOTYP3YXT3P3EKCEL5FTYQNRSA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.646095778 (493h17m11.517178813s) [noise=true , address=145.79.227.10:4402 , piece_id=WLSY7UMISW4KS74K6QX5WL3K5FPX3APLEZXZYWFKCAR5CTPETW3Q, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.646499165 (493h17m11.51758221s) [noise=true , address=145.79.227.12:4767 , piece_id=BAMCB3OPKUOFWJOLCZKLOSGS4V5ZU6VXY6YI37UCDQBMSIKCTEBQ, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.646905189 (493h17m11.517988224s) [noise=true , address=145.79.227.46:4602 , piece_id=RFEQYLZ5IKPEE6G2URQQZLBTLVYIJLZLREUISCPYVJXNXFVSZOHA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.647052429 (493h17m11.518135474s) [noise=true , address=145.79.227.17:4970 , piece_id=VBA5Z34H3ENLKKF5U3BUFXRN3TODH3N6EA2GPXVETZSQQN53PYGQ, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.647302345 (493h17m11.51838539s) [noise=true , address=145.79.227.41:4085 , piece_id=CCQUJOZZXLLTKJJFXSIL7ERDJIIPXLF4D5UV7D67O7YRZFPZMV2A, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.647612465 (493h17m11.51869551s) [noise=true , address=145.79.227.16:4910 , piece_id=LSTHPP5EPYRTS6OLJ63S7JQ2LA2NUEIEWJGUY4C3LF3LTZHUCD7Q, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.648183694 (493h17m11.519266739s) [noise=true , address=145.79.227.47:4266 , piece_id=XWN34ED3GUVXA3PPTX5QU4KHR7CBBC745V44S3W57FBGRQ5I4HJA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.649790443 (493h17m11.520873488s) [noise=true , address=45.91.116.13:4153 , piece_id=ZK6U7BL3NYLN6Z5ZLS5BGUX5WEVFRSMVCN7ZRE35TOLTNTH46FDA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:13.268673497 (493h17m13.139756543s) [upload=432954]: Next returned nil segment Dec 26 04:46:13.268688636 (493h17m13.139771681s) [upload=432954]: Waiting for error group managing segments... Dec 26 04:46:13.647888619 (493h17m13.518971664s) [noise=true , address=145.79.227.12:4767 , piece_id=BAMCB3OPKUOFWJOLCZKLOSGS4V5ZU6VXY6YI37UCDQBMSIKCTEBQ, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.650591886 (493h17m13.521674921s) [noise=true , address=145.79.227.35:4483 , piece_id=QNQAWWDM2FOPRQ45KCY466RB5WW3CZMS2OADEXADNWYCL3HGE2DA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.651254949 (493h17m13.522337994s) [noise=true , address=145.79.227.45:4230 , piece_id=QCJQZ4FYVPORIDUACXOUM6BNY55AAKLI7RA7XAPG3G5DDSAXZNNA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.651861003 (493h17m13.522944038s) [noise=true , address=145.79.227.49:4338 , piece_id=O5P7VKX3ILX2OMI4U3CYS2CDGWFILNQ4S3IMGU45MI4LNEEOLVMA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.653880118 (493h17m13.524963153s) [noise=true , address=145.79.227.41:4085 , piece_id=CCQUJOZZXLLTKJJFXSIL7ERDJIIPXLF4D5UV7D67O7YRZFPZMV2A, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.654533863 (493h17m13.525616898s) [noise=true , address=145.79.227.46:4602 , piece_id=RFEQYLZ5IKPEE6G2URQQZLBTLVYIJLZLREUISCPYVJXNXFVSZOHA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.655418407 (493h17m13.526501442s) [noise=true , address=145.79.227.37:4566 , piece_id=QUYPK2QBYVUT6FCVJ3QNHJQZYW7KKJFH7TRB7MWGG5HFKQOFGVYA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.657324486 (493h17m13.528407521s) [noise=true , address=145.79.227.10:4402 , piece_id=WLSY7UMISW4KS74K6QX5WL3K5FPX3APLEZXZYWFKCAR5CTPETW3Q, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.660265606 (493h17m13.531348651s) [noise=true , address=145.79.227.11:4732 , piece_id=BYOYWN56YERGV4T72GZTAZAGZ4PKNVO76AINW5APTS37NNWQWLIA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.663478735 (493h17m13.53456177s) [noise=true , address=145.79.227.17:4970 , piece_id=VBA5Z34H3ENLKKF5U3BUFXRN3TODH3N6EA2GPXVETZSQQN53PYGQ, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.670422557 (493h17m13.541505602s) [noise=true , address=145.79.227.47:4266 , piece_id=XWN34ED3GUVXA3PPTX5QU4KHR7CBBC745V44S3W57FBGRQ5I4HJA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.674293307 (493h17m13.545376342s) [noise=true , address=145.79.227.5:4221 , piece_id=33PLGGV5LFV2MQ5NIAGFDW3NZEDBUOPREN2EGNVKUYPNM76E4RAQ, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.675494666 (493h17m13.546577711s) [noise=true , address=145.79.227.4:4185 , piece_id=4OC43GK3EPEXGNVDBVCLUTQWWKVOTYP3YXT3P3EKCEL5FTYQNRSA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.675504344 (493h17m13.546587379s) [seg_pos={0 0}, upload=432954]: Segment reached optimal threshold of 13 pieces. Dec 26 04:46:13.675660802 (493h17m13.546743847s) [noise=true , address=45.91.116.5:4299 , piece_id=EWLF23X32SDKY6KVSU4PUCZEYXWBJQW2OZCA57SACRQSLMUUYWKQ, seg_pos={0 0}, upload=432954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 04:46:13.675667525 (493h17m13.54675056s) [noise=true , address=45.91.116.5:4299 , piece_id=EWLF23X32SDKY6KVSU4PUCZEYXWBJQW2OZCA57SACRQSLMUUYWKQ, seg_pos={0 0}, upload=432954]: Long tail cancellation due to optimal threshold Dec 26 04:46:13.675867456 (493h17m13.546950501s) [noise=true , address=145.79.227.16:4910 , piece_id=LSTHPP5EPYRTS6OLJ63S7JQ2LA2NUEIEWJGUY4C3LF3LTZHUCD7Q, seg_pos={0 0}, upload=432954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 26 04:46:13.675874379 (493h17m13.546957424s) [noise=true , address=145.79.227.16:4910 , piece_id=LSTHPP5EPYRTS6OLJ63S7JQ2LA2NUEIEWJGUY4C3LF3LTZHUCD7Q, seg_pos={0 0}, upload=432954]: Long tail cancellation due to optimal threshold Dec 26 04:46:13.675960001 (493h17m13.547043036s) [noise=true , address=45.91.116.13:4153 , piece_id=ZK6U7BL3NYLN6Z5ZLS5BGUX5WEVFRSMVCN7ZRE35TOLTNTH46FDA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 04:46:13.675965011 (493h17m13.547048056s) [noise=true , address=45.91.116.13:4153 , piece_id=ZK6U7BL3NYLN6Z5ZLS5BGUX5WEVFRSMVCN7ZRE35TOLTNTH46FDA, seg_pos={0 0}, upload=432954]: Long tail cancellation due to optimal threshold Dec 26 04:46:13.675980591 (493h17m13.547063636s) [seg_pos={0 0}, upload=432954]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:46:13.676048029 (493h17m13.547131064s) [upload=432954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:46:13.676053339 (493h17m13.547136384s) [upload=432954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:46:13.955881701 (493h17m13.826964736s) [upload=432954]: Done uploading segments... Dec 26 04:46:13.955908562 (493h17m13.826991617s) [upload=432954]: Upload finished. err: <nil> Duration: 6.031103178s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.342953315s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:49:23 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.342980967s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.046929713s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:48:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.046955093s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 32.948356712s trace: https://jaeger.dev.storj.io/trace/3025aa9e97d9eb91 FAIL (2025-12-26 04:48:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m8.672938647s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.802601632 (492h52m8.673684677s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803110081 (492h52m8.674193126s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.803583533 (492h52m8.674666568s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804082522 (492h52m8.675165577s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.804817221 (492h52m8.675900266s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.805380093 (492h52m8.676463158s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806020182 (492h52m8.677103227s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.806108530 (492h52m8.677191575s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807079158 (492h52m8.678162193s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.807775344 (492h52m8.678858389s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808355900 (492h52m8.679438945s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.808939461 (492h52m8.680022506s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.809560894 (492h52m8.680643939s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.810309420 (492h52m8.681392465s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:08.814096872 (492h52m8.685179917s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Uploading piece... Dec 26 04:21:10.274244135 (492h52m10.14532718s) [upload=432577]: Next returned nil segment Dec 26 04:21:10.274358442 (492h52m10.145441487s) [upload=432577]: Waiting for error group managing segments... Dec 26 04:21:10.679035801 (492h52m10.550118846s) [noise=true , address=145.79.227.12:4770 , piece_id=OILRR2BTPASWRSY4SID7K4GLVZ7CS5TCUNFVYL7HWH4IZ7QX7SJA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.714666859 (492h52m10.585749894s) [noise=true , address=145.79.227.7:4318 , piece_id=IZM6KHIHOIFYF3T4W7V7SYPLLIOZKXZDRUPXJHH55QHKOW7ICNEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.719697578 (492h52m10.590780623s) [noise=true , address=145.79.227.43:4177 , piece_id=GAR4OZ7SPTZ4REI2QILQEANJOOGM4BWTQF6HEKRAVF2PPODOZIWA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.726102654 (492h52m10.597185699s) [noise=true , address=145.79.227.34:4913 , piece_id=CRS2OHQY55TII6YF3W7XTJYINU3EDYRK4YLDNUMUTU4OXUQAESXA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.887615976 (492h52m10.758699011s) [noise=true , address=77.90.31.11:4578 , piece_id=2PHEV32BEYSTIVGWRWQYHE4USYFZO2XFNRPNXX6PDL64F3OHSAUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.895646197 (492h52m10.766729242s) [noise=true , address=45.91.116.8:4058 , piece_id=ENZSOUIQQNSFO7SVPDYEFTOTSR3WJMXGDTVRPVFITLXNL4CYMALQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.902034461 (492h52m10.773117496s) [noise=true , address=77.90.31.18:4809 , piece_id=XGFM4K6LQWGHCNTN3MZWHZZSH4OIP3IPFGGEI4EFWH24MIOSB64Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.932217795 (492h52m10.80330084s) [noise=true , address=45.91.116.12:4252 , piece_id=GLPTLWRBPFSIZY737H4JEBWPVB3BVIPBUVFMS6ZG4GTHMGAYIA6Q, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.947318980 (492h52m10.818402015s) [noise=true , address=77.90.31.13:4650 , piece_id=UYGZNV4GR7L6WEFT3Z7CXKK4WGXZKIVXQKBU2ERQXLGLWBXRCTUQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.950375581 (492h52m10.821458626s) [noise=true , address=77.90.31.15:4711 , piece_id=GHCH3OBTJV4GW32OSYVEKAPWFZGZO6RNEJVLM3KYVYPDV65276LA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.968869566 (492h52m10.839952601s) [noise=true , address=145.79.227.33:4892 , piece_id=HNLNJLUERHE2LKH77YPNIJKVTU2X6M35KYR3V2ZYIBPXYA3KLSEQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:10.974048438 (492h52m10.845131483s) [noise=true , address=77.90.31.14:4682 , piece_id=HKJ7HEXHDNPVXGCRWOWBLQQVJ6WCZ3VMSIXSBGRZKUPKCFZEAKTA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.568745408 (492h52m11.439828443s) [noise=true , address=145.79.227.30:4786 , piece_id=JXKDRQRXPLVQQN3R65FBOTBIHZ7PW5KDOIPVAFWSDHK6AKDA4YNA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669213026 (492h52m11.540296072s) [noise=true , address=145.79.227.44:4210 , piece_id=VCS3RFS7SLBWGQMBREITJA3WR2QH5B5MEF5TSRAUTWI3VHJNRSCA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: <nil> Dec 26 04:21:11.669239968 (492h52m11.540323013s) [seg_pos={0 0}, upload=432577]: Segment reached optimal threshold of 14 pieces. Dec 26 04:21:11.669499321 (492h52m11.540582387s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60336->64.49.13.169:4711: use of closed network connection Dec 26 04:21:11.669509792 (492h52m11.540592857s) [noise=true , address=64.49.13.169:4711 , piece_id=BFTE2JION6PUNXIGHRCKQNZ4FN2C6T3XC2L25BOXWWKMJ33V2UXQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669610564 (492h52m11.540693619s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669632555 (492h52m11.54071563s) [noise=true , address=77.90.31.19:4858 , piece_id=CVCCZD4Q47ZW4HSC2LM4U42DRCAJBO6EMIHTRU5ZQUXNII66R6VA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669854298 (492h52m11.540937343s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669861522 (492h52m11.540944567s) [noise=true , address=77.90.31.21:4901 , piece_id=DGWWLVPZOVTB7DVZVZJ23NM6VLI63BSSS7LLTJKOF36RJ5SFUYMQ, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669935953 (492h52m11.541018988s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6J9mfHN8XiwHLAMaJVjAnaLqbx1oUoQSa5AkmcA9TqEpPnSJ): protocol: expected piece hash; context canceled; EOF Dec 26 04:21:11.669947755 (492h52m11.5410308s) [noise=true , address=145.79.227.40:4058 , piece_id=URTEWSL5C2C746RJ3QUUYXZXY6J4OGD22MQZL2BRTJKXZKQYZKBA, seg_pos={0 0}, upload=432577]: Long tail cancellation due to optimal threshold Dec 26 04:21:11.669975398 (492h52m11.541058433s) [seg_pos={0 0}, upload=432577]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 04:21:11.670066051 (492h52m11.541149096s) [upload=432577]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:21:11.670072994 (492h52m11.541156039s) [upload=432577]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:21:11.956307857 (492h52m11.827390902s) [upload=432577]: Done uploading segments... Dec 26 04:21:11.956340630 (492h52m11.827423665s) [upload=432577]: Upload finished. err: <nil> Duration: 32.948381842s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.216063875s trace: https://jaeger.dev.storj.io/trace/31feed8c86845b37 FAIL (2025-12-26 04:48:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 6.216093872s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........................X..X..X.X....XX.X.X.X.XX.X......X...] duration: 5.959003458s trace: https://jaeger.dev.storj.io/trace/7fd1f6e029d2997a FAIL (2025-12-26 04:48:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, Piece ID: 33PLGGV5LFV2MQ5NIAGFDW3NZEDBUOPREN2EGNVKUYPNM76E4RAQ) context deadline exceeded; piecestore: (Node ID: 1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, Piece ID: 4OC43GK3EPEXGNVDBVCLUTQWWKVOTYP3YXT3P3EKCEL5FTYQNRSA) context deadline exceeded; piecestore: (Node ID: 1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d, Piece ID: RFEQYLZ5IKPEE6G2URQQZLBTLVYIJLZLREUISCPYVJXNXFVSZOHA) context deadline exceeded; piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: VBA5Z34H3ENLKKF5U3BUFXRN3TODH3N6EA2GPXVETZSQQN53PYGQ) context deadline exceeded; piecestore: (Node ID: 1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, Piece ID: QNQAWWDM2FOPRQ45KCY466RB5WW3CZMS2OADEXADNWYCL3HGE2DA) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: O5P7VKX3ILX2OMI4U3CYS2CDGWFILNQ4S3IMGU45MI4LNEEOLVMA) context deadline exceeded ======LOGS====== Dec 26 04:46:11.403823619 (493h17m11.274906665s) [upload=432954]: Starting upload Dec 26 04:46:11.403884125 (493h17m11.27496717s) [upload=432954]: Done starting upload Dec 26 04:46:11.404071982 (493h17m11.275155017s) [upload=432954]: Uploading segments... Dec 26 04:46:11.404120314 (493h17m11.275203349s) [upload=432954]: Got next segment. Inline: false Dec 26 04:46:11.404146584 (493h17m11.275229629s) [upload=432954]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 04:46:11.404151002 (493h17m11.275234048s) [upload=432954]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 04:46:11.643720875 (493h17m11.51480391s) [seg_pos={0 0}, upload=432954]: Begin upload segment... Dec 26 04:46:11.643771943 (493h17m11.514854978s) [seg_pos={0 0}, upload=432954]: Done begin upload segment. Dec 26 04:46:11.643815044 (493h17m11.514898079s) [noise=true , address=145.79.227.45:4230 , piece_id=QCJQZ4FYVPORIDUACXOUM6BNY55AAKLI7RA7XAPG3G5DDSAXZNNA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.643864538 (493h17m11.514947583s) [noise=true , address=145.79.227.35:4483 , piece_id=QNQAWWDM2FOPRQ45KCY466RB5WW3CZMS2OADEXADNWYCL3HGE2DA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.644634676 (493h17m11.515717711s) [noise=true , address=145.79.227.49:4338 , piece_id=O5P7VKX3ILX2OMI4U3CYS2CDGWFILNQ4S3IMGU45MI4LNEEOLVMA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.644880443 (493h17m11.515963489s) [noise=true , address=145.79.227.37:4566 , piece_id=QUYPK2QBYVUT6FCVJ3QNHJQZYW7KKJFH7TRB7MWGG5HFKQOFGVYA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.645033354 (493h17m11.516116389s) [noise=true , address=145.79.227.11:4732 , piece_id=BYOYWN56YERGV4T72GZTAZAGZ4PKNVO76AINW5APTS37NNWQWLIA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.645379403 (493h17m11.516462438s) [noise=true , address=145.79.227.5:4221 , piece_id=33PLGGV5LFV2MQ5NIAGFDW3NZEDBUOPREN2EGNVKUYPNM76E4RAQ, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.645466308 (493h17m11.516549343s) [noise=true , address=45.91.116.5:4299 , piece_id=EWLF23X32SDKY6KVSU4PUCZEYXWBJQW2OZCA57SACRQSLMUUYWKQ, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.645786348 (493h17m11.516869383s) [noise=true , address=145.79.227.4:4185 , piece_id=4OC43GK3EPEXGNVDBVCLUTQWWKVOTYP3YXT3P3EKCEL5FTYQNRSA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.646095778 (493h17m11.517178813s) [noise=true , address=145.79.227.10:4402 , piece_id=WLSY7UMISW4KS74K6QX5WL3K5FPX3APLEZXZYWFKCAR5CTPETW3Q, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.646499165 (493h17m11.51758221s) [noise=true , address=145.79.227.12:4767 , piece_id=BAMCB3OPKUOFWJOLCZKLOSGS4V5ZU6VXY6YI37UCDQBMSIKCTEBQ, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.646905189 (493h17m11.517988224s) [noise=true , address=145.79.227.46:4602 , piece_id=RFEQYLZ5IKPEE6G2URQQZLBTLVYIJLZLREUISCPYVJXNXFVSZOHA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.647052429 (493h17m11.518135474s) [noise=true , address=145.79.227.17:4970 , piece_id=VBA5Z34H3ENLKKF5U3BUFXRN3TODH3N6EA2GPXVETZSQQN53PYGQ, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.647302345 (493h17m11.51838539s) [noise=true , address=145.79.227.41:4085 , piece_id=CCQUJOZZXLLTKJJFXSIL7ERDJIIPXLF4D5UV7D67O7YRZFPZMV2A, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.647612465 (493h17m11.51869551s) [noise=true , address=145.79.227.16:4910 , piece_id=LSTHPP5EPYRTS6OLJ63S7JQ2LA2NUEIEWJGUY4C3LF3LTZHUCD7Q, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.648183694 (493h17m11.519266739s) [noise=true , address=145.79.227.47:4266 , piece_id=XWN34ED3GUVXA3PPTX5QU4KHR7CBBC745V44S3W57FBGRQ5I4HJA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:11.649790443 (493h17m11.520873488s) [noise=true , address=45.91.116.13:4153 , piece_id=ZK6U7BL3NYLN6Z5ZLS5BGUX5WEVFRSMVCN7ZRE35TOLTNTH46FDA, seg_pos={0 0}, upload=432954]: Uploading piece... Dec 26 04:46:13.268673497 (493h17m13.139756543s) [upload=432954]: Next returned nil segment Dec 26 04:46:13.268688636 (493h17m13.139771681s) [upload=432954]: Waiting for error group managing segments... Dec 26 04:46:13.647888619 (493h17m13.518971664s) [noise=true , address=145.79.227.12:4767 , piece_id=BAMCB3OPKUOFWJOLCZKLOSGS4V5ZU6VXY6YI37UCDQBMSIKCTEBQ, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.650591886 (493h17m13.521674921s) [noise=true , address=145.79.227.35:4483 , piece_id=QNQAWWDM2FOPRQ45KCY466RB5WW3CZMS2OADEXADNWYCL3HGE2DA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.651254949 (493h17m13.522337994s) [noise=true , address=145.79.227.45:4230 , piece_id=QCJQZ4FYVPORIDUACXOUM6BNY55AAKLI7RA7XAPG3G5DDSAXZNNA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.651861003 (493h17m13.522944038s) [noise=true , address=145.79.227.49:4338 , piece_id=O5P7VKX3ILX2OMI4U3CYS2CDGWFILNQ4S3IMGU45MI4LNEEOLVMA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.653880118 (493h17m13.524963153s) [noise=true , address=145.79.227.41:4085 , piece_id=CCQUJOZZXLLTKJJFXSIL7ERDJIIPXLF4D5UV7D67O7YRZFPZMV2A, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.654533863 (493h17m13.525616898s) [noise=true , address=145.79.227.46:4602 , piece_id=RFEQYLZ5IKPEE6G2URQQZLBTLVYIJLZLREUISCPYVJXNXFVSZOHA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.655418407 (493h17m13.526501442s) [noise=true , address=145.79.227.37:4566 , piece_id=QUYPK2QBYVUT6FCVJ3QNHJQZYW7KKJFH7TRB7MWGG5HFKQOFGVYA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.657324486 (493h17m13.528407521s) [noise=true , address=145.79.227.10:4402 , piece_id=WLSY7UMISW4KS74K6QX5WL3K5FPX3APLEZXZYWFKCAR5CTPETW3Q, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.660265606 (493h17m13.531348651s) [noise=true , address=145.79.227.11:4732 , piece_id=BYOYWN56YERGV4T72GZTAZAGZ4PKNVO76AINW5APTS37NNWQWLIA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.663478735 (493h17m13.53456177s) [noise=true , address=145.79.227.17:4970 , piece_id=VBA5Z34H3ENLKKF5U3BUFXRN3TODH3N6EA2GPXVETZSQQN53PYGQ, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.670422557 (493h17m13.541505602s) [noise=true , address=145.79.227.47:4266 , piece_id=XWN34ED3GUVXA3PPTX5QU4KHR7CBBC745V44S3W57FBGRQ5I4HJA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.674293307 (493h17m13.545376342s) [noise=true , address=145.79.227.5:4221 , piece_id=33PLGGV5LFV2MQ5NIAGFDW3NZEDBUOPREN2EGNVKUYPNM76E4RAQ, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.675494666 (493h17m13.546577711s) [noise=true , address=145.79.227.4:4185 , piece_id=4OC43GK3EPEXGNVDBVCLUTQWWKVOTYP3YXT3P3EKCEL5FTYQNRSA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: <nil> Dec 26 04:46:13.675504344 (493h17m13.546587379s) [seg_pos={0 0}, upload=432954]: Segment reached optimal threshold of 13 pieces. Dec 26 04:46:13.675660802 (493h17m13.546743847s) [noise=true , address=45.91.116.5:4299 , piece_id=EWLF23X32SDKY6KVSU4PUCZEYXWBJQW2OZCA57SACRQSLMUUYWKQ, seg_pos={0 0}, upload=432954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 26 04:46:13.675667525 (493h17m13.54675056s) [noise=true , address=45.91.116.5:4299 , piece_id=EWLF23X32SDKY6KVSU4PUCZEYXWBJQW2OZCA57SACRQSLMUUYWKQ, seg_pos={0 0}, upload=432954]: Long tail cancellation due to optimal threshold Dec 26 04:46:13.675867456 (493h17m13.546950501s) [noise=true , address=145.79.227.16:4910 , piece_id=LSTHPP5EPYRTS6OLJ63S7JQ2LA2NUEIEWJGUY4C3LF3LTZHUCD7Q, seg_pos={0 0}, upload=432954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: expected piece hash; context canceled; EOF Dec 26 04:46:13.675874379 (493h17m13.546957424s) [noise=true , address=145.79.227.16:4910 , piece_id=LSTHPP5EPYRTS6OLJ63S7JQ2LA2NUEIEWJGUY4C3LF3LTZHUCD7Q, seg_pos={0 0}, upload=432954]: Long tail cancellation due to optimal threshold Dec 26 04:46:13.675960001 (493h17m13.547043036s) [noise=true , address=45.91.116.13:4153 , piece_id=ZK6U7BL3NYLN6Z5ZLS5BGUX5WEVFRSMVCN7ZRE35TOLTNTH46FDA, seg_pos={0 0}, upload=432954]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 04:46:13.675965011 (493h17m13.547048056s) [noise=true , address=45.91.116.13:4153 , piece_id=ZK6U7BL3NYLN6Z5ZLS5BGUX5WEVFRSMVCN7ZRE35TOLTNTH46FDA, seg_pos={0 0}, upload=432954]: Long tail cancellation due to optimal threshold Dec 26 04:46:13.675980591 (493h17m13.547063636s) [seg_pos={0 0}, upload=432954]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 04:46:13.676048029 (493h17m13.547131064s) [upload=432954]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 04:46:13.676053339 (493h17m13.547136384s) [upload=432954]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 04:46:13.955881701 (493h17m13.826964736s) [upload=432954]: Done uploading segments... Dec 26 04:46:13.955908562 (493h17m13.826991617s) [upload=432954]: Upload finished. err: <nil> Duration: 5.959024577s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)