Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.......X.X......X..............X..X.X...X.XX..XX.X.......XXX] duration: 5.000826966s trace: https://jaeger.dev.storj.io/trace/43ee03c677c762ef FAIL (2025-12-27 13:15:42 UTC) orbiter warning: context deadline exceeded Duration: 5.000846945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X.X..XX.X......X..X..XXX...........XX..X...XX......X..X..] duration: 6.124891749s trace: https://jaeger.dev.storj.io/trace/66a858bc169820a3 FAIL (2025-12-27 13:15:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 13:14:00.930597344 (525h45m0.80168039s) [upload=461170]: Starting upload Dec 27 13:14:00.930690642 (525h45m0.801773718s) [upload=461170]: Done starting upload Dec 27 13:14:00.930803036 (525h45m0.801886081s) [upload=461170]: Uploading segments... Dec 27 13:14:00.930978100 (525h45m0.802061145s) [upload=461170]: Got next segment. Inline: false Dec 27 13:14:00.931020871 (525h45m0.802103906s) [upload=461170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:14:00.931027663 (525h45m0.802110709s) [upload=461170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:14:01.176830284 (525h45m1.047913319s) [seg_pos={0 0}, upload=461170]: Begin upload segment... Dec 27 13:14:01.176905156 (525h45m1.047988191s) [seg_pos={0 0}, upload=461170]: Done begin upload segment. Dec 27 13:14:01.177015857 (525h45m1.048098902s) [noise=true , address=77.90.31.10:4517 , piece_id=XBIASFXOPMDKDK7AE3HQRHXVDNJGG5A6EEVNU7MCJ7MJDPHQ2NTQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.177105558 (525h45m1.048188603s) [noise=true , address=145.79.227.45:4231 , piece_id=QPZWGHITJ4TH6AAOKEZCI3UFGNERNHQSEDFQEDDQNSMLAZNTUXZA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.177668801 (525h45m1.048751846s) [noise=true , address=145.79.227.37:4566 , piece_id=26XNZ4FAC7MT2WYBQSGWOZT3AQ5GFDNSHWOXRJULY4UYREYADS3A, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.178230680 (525h45m1.049313715s) [noise=true , address=145.79.227.36:4518 , piece_id=53D2BGXYKZPOH77NGYZY64IGQ3JWQJEODCIUYNHP7FZ52EFL7FXA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.179606671 (525h45m1.050689716s) [noise=true , address=77.90.31.14:4661 , piece_id=YIKZB3QDUYR2Q5VRPEX3BSK2IVCX4UQGYEXQUG4MQOEOA2XKNTFQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.180250296 (525h45m1.051333341s) [noise=true , address=77.90.31.11:4554 , piece_id=QDGSXF6Y6KN7HF5UKEFNWBMB7SQ6TMGRWMANLCU5EZ7BMAYLM4DQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.181620325 (525h45m1.052703371s) [noise=true , address=77.90.31.12:4590 , piece_id=HG6DWEFAOLXYJ36YBBJQLM546AZ3IEMZX3AVPZABFTF4KJC7BMRA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.181920326 (525h45m1.053003371s) [noise=true , address=45.91.116.12:4226 , piece_id=XPTZ55NVVT67SMQ24CDVVNKVCT774QEUNCGBDAIGVEYMIRL2SXRQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.183146732 (525h45m1.054229777s) [noise=true , address=145.79.227.42:4123 , piece_id=G4K3B5CGIN4EF5DDAXAQBNTA74QRX76BMG3VOC4ONBM72FSWJKZA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.183911218 (525h45m1.054994253s) [noise=true , address=77.90.31.21:4880 , piece_id=DL3S24B5QP2JSML25CGUAOXNZP3KJWYAV7JY3IDK67PRQ6O727GQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.185980387 (525h45m1.057063443s) [noise=true , address=77.90.31.19:4841 , piece_id=PQPFCVGQ42FTXXOV7MSOREIZK6QPFAUCYDPVAIL5HKXJDLL52SWQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.188163595 (525h45m1.05924665s) [noise=true , address=145.79.227.39:4015 , piece_id=IMVEMCHUQWHJPR4ACRB2Z7IDZXM4ZGW4RWYSC66TXTCRI3YLEQSA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.189639545 (525h45m1.06072259s) [noise=true , address=145.79.227.11:4732 , piece_id=NNSTTBHYAQJPPBXECUZE4T24JQBFZSFHGWZ3ERRR5PAJ2I3SUNZA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.191979010 (525h45m1.063062055s) [noise=true , address=145.79.227.2:4114 , piece_id=Q2MWW6QINH2UIU47NLBZOECNECDLQD5ZMT7JSHJPFB733S6F73PA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.194386303 (525h45m1.065469348s) [noise=true , address=145.79.227.46:4603 , piece_id=TZ7FWPRWMQRIPAOBD47SFVDSMHOET5KFZ63UAY6XEIH25BWWNRKA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.195559157 (525h45m1.066642202s) [noise=true , address=145.79.227.51:4410 , piece_id=SEZMCOVJXDXC5IWB26N6TMHACJX467SOWQSNPZ2F3WVATQUISWGQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:02.799598243 (525h45m2.670681299s) [upload=461170]: Next returned nil segment Dec 27 13:14:02.799619344 (525h45m2.670702389s) [upload=461170]: Waiting for error group managing segments... Dec 27 13:14:03.185957448 (525h45m3.057040493s) [noise=true , address=145.79.227.42:4123 , piece_id=G4K3B5CGIN4EF5DDAXAQBNTA74QRX76BMG3VOC4ONBM72FSWJKZA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.188334494 (525h45m3.059417539s) [noise=true , address=145.79.227.2:4114 , piece_id=Q2MWW6QINH2UIU47NLBZOECNECDLQD5ZMT7JSHJPFB733S6F73PA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.189580988 (525h45m3.060664033s) [noise=true , address=145.79.227.36:4518 , piece_id=53D2BGXYKZPOH77NGYZY64IGQ3JWQJEODCIUYNHP7FZ52EFL7FXA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.196858015 (525h45m3.06794106s) [noise=true , address=145.79.227.39:4015 , piece_id=IMVEMCHUQWHJPR4ACRB2Z7IDZXM4ZGW4RWYSC66TXTCRI3YLEQSA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.199856634 (525h45m3.070939679s) [noise=true , address=145.79.227.37:4566 , piece_id=26XNZ4FAC7MT2WYBQSGWOZT3AQ5GFDNSHWOXRJULY4UYREYADS3A, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.210819920 (525h45m3.081902976s) [noise=true , address=145.79.227.46:4603 , piece_id=TZ7FWPRWMQRIPAOBD47SFVDSMHOET5KFZ63UAY6XEIH25BWWNRKA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.210854757 (525h45m3.081937802s) [noise=true , address=145.79.227.45:4231 , piece_id=QPZWGHITJ4TH6AAOKEZCI3UFGNERNHQSEDFQEDDQNSMLAZNTUXZA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.217484562 (525h45m3.088567607s) [noise=true , address=145.79.227.51:4410 , piece_id=SEZMCOVJXDXC5IWB26N6TMHACJX467SOWQSNPZ2F3WVATQUISWGQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.219463480 (525h45m3.090546525s) [noise=true , address=145.79.227.11:4732 , piece_id=NNSTTBHYAQJPPBXECUZE4T24JQBFZSFHGWZ3ERRR5PAJ2I3SUNZA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.395263393 (525h45m3.266346468s) [noise=true , address=77.90.31.11:4554 , piece_id=QDGSXF6Y6KN7HF5UKEFNWBMB7SQ6TMGRWMANLCU5EZ7BMAYLM4DQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.433922256 (525h45m3.305005301s) [noise=true , address=77.90.31.21:4880 , piece_id=DL3S24B5QP2JSML25CGUAOXNZP3KJWYAV7JY3IDK67PRQ6O727GQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.442870405 (525h45m3.31395345s) [noise=true , address=77.90.31.12:4590 , piece_id=HG6DWEFAOLXYJ36YBBJQLM546AZ3IEMZX3AVPZABFTF4KJC7BMRA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.451919566 (525h45m3.323002611s) [noise=true , address=77.90.31.19:4841 , piece_id=PQPFCVGQ42FTXXOV7MSOREIZK6QPFAUCYDPVAIL5HKXJDLL52SWQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.451970804 (525h45m3.323053849s) [seg_pos={0 0}, upload=461170]: Segment reached optimal threshold of 13 pieces. Dec 27 13:14:03.452400842 (525h45m3.323483887s) [noise=true , address=77.90.31.10:4517 , piece_id=XBIASFXOPMDKDK7AE3HQRHXVDNJGG5A6EEVNU7MCJ7MJDPHQ2NTQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:03.452425550 (525h45m3.323508605s) [noise=true , address=77.90.31.10:4517 , piece_id=XBIASFXOPMDKDK7AE3HQRHXVDNJGG5A6EEVNU7MCJ7MJDPHQ2NTQ, seg_pos={0 0}, upload=461170]: Long tail cancellation due to optimal threshold Dec 27 13:14:03.452608748 (525h45m3.323691793s) [noise=true , address=77.90.31.14:4661 , piece_id=YIKZB3QDUYR2Q5VRPEX3BSK2IVCX4UQGYEXQUG4MQOEOA2XKNTFQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:03.452618477 (525h45m3.323701522s) [noise=true , address=77.90.31.14:4661 , piece_id=YIKZB3QDUYR2Q5VRPEX3BSK2IVCX4UQGYEXQUG4MQOEOA2XKNTFQ, seg_pos={0 0}, upload=461170]: Long tail cancellation due to optimal threshold Dec 27 13:14:03.452714790 (525h45m3.323797825s) [noise=true , address=45.91.116.12:4226 , piece_id=XPTZ55NVVT67SMQ24CDVVNKVCT774QEUNCGBDAIGVEYMIRL2SXRQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:03.452724558 (525h45m3.323807594s) [noise=true , address=45.91.116.12:4226 , piece_id=XPTZ55NVVT67SMQ24CDVVNKVCT774QEUNCGBDAIGVEYMIRL2SXRQ, seg_pos={0 0}, upload=461170]: Long tail cancellation due to optimal threshold Dec 27 13:14:03.452788731 (525h45m3.323871766s) [seg_pos={0 0}, upload=461170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:14:03.452885345 (525h45m3.32396839s) [upload=461170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:14:03.452893611 (525h45m3.323976646s) [upload=461170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:14:03.747125409 (525h45m3.618208454s) [upload=461170]: Done uploading segments... Dec 27 13:14:03.747154284 (525h45m3.618237329s) [upload=461170]: Upload finished. err: <nil> Duration: 6.124919272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.............................X...X] duration: 1m0.002224731s trace: https://jaeger.dev.storj.io/trace/3fff9d1180db5d69 FAIL (2025-12-27 13:15:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, Piece ID: UEKHT3CPCVS5FKTHTXI7C7OPLJCDSWWJEVZBKKCGXOVJQUSUOYTQ) context deadline exceeded; piecestore: (Node ID: 123QEF1LQm8nDCTjZvq51BjSMTYwHjhj2ajZd7bCRjiWJVLGXyS, Piece ID: 4IOSYSIGU2FGHLMUUS5CTYAK3IVKC5UKEPFKTOOZJLUCHU46QVOQ) context deadline exceeded; piecestore: (Node ID: 1UydYeC9AEPHMXQZGaDhgS2vNguG4umL5nnBLpty2L3zeQAvLu, Piece ID: 2NBPJXVK52L2M7HNYEH3V5XMA5TJ3O6C27PON23W5QLXYU3S6PAA) context deadline exceeded ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=WYPVV5ND6727W3RUHKVEBW7TY2SLK44OHGIWKNQS3Z7VRA7R6BHA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.313890357 (525h45m29.184973402s) [noise=true , address=77.90.31.19:4874 , piece_id=SXRHPQ6ZKSYS7XMDFM4NM444R3UD77XDCZCLDJGYNVXMC5ZSYWAQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314231116 (525h45m29.185314151s) [noise=true , address=45.91.116.5:4329 , piece_id=4IOSYSIGU2FGHLMUUS5CTYAK3IVKC5UKEPFKTOOZJLUCHU46QVOQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314397303 (525h45m29.185480338s) [noise=true , address=77.90.31.15:4712 , piece_id=PGSHIFGJWOYKM6E7PWUDTUZP6S32PETNZLD73GL3ZMIZP4REYQMA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314590300 (525h45m29.185673345s) [noise=true , address=77.90.31.16:4759 , piece_id=GG44CNIRWLO5NTIZPNEEQY522WZQBRKZYRKW6QMICCNKF6FPL45A, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314728854 (525h45m29.185811889s) [noise=true , address=145.79.227.40:4062 , piece_id=PN5ZDTWCSQEQXCRG7Q2KWJ35JA26CIVQCEXIPYH7MSEFCW55JAXA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.314946217 (525h45m29.186029262s) [noise=true , address=145.79.227.48:4311 , piece_id=JPY7E33FC6AWCECY6IQEXPMRUSCR6EDJXRFARMU7ZWXS5D7RRVUQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.315053011 (525h45m29.186136046s) [noise=true , address=64.49.13.169:4710 , piece_id=TIN6WHF6JQRHYT5BYZISPGW6D2QUVYZ4RXVJHBPVZEHJQXFXY2FQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.315309129 (525h45m29.186392174s) [noise=true , address=45.91.116.10:4089 , piece_id=63NTMXR7WBHBQ52HJUCB2SE5NBCPSZVBECHIKD3LPYFTM7KA3ZOQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.315374393 (525h45m29.186457428s) [noise=true , address=45.91.116.4:4964 , piece_id=PS273B6BPO4O6J6G4LQLCVRAOKP67PKW4NZRR5FPN5RPCB2MEQQQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.315805594 (525h45m29.186888629s) [noise=true , address=145.79.227.44:4225 , piece_id=45QAKVCQ5FPOLE5DRCG2WSFIOYO7C7NH4HTYHSQILOUFM7EFXO2Q, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.316139150 (525h45m29.187222185s) [noise=true , address=77.90.31.21:4894 , piece_id=GR5PRAIVWGPPSNDO2IKB7VN65UBASLHZELBLHORDGYOQA4LWZRSA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.316474948 (525h45m29.187557983s) [noise=true , address=145.79.227.30:4768 , piece_id=DQ23KUHNR6GG5RA44UOSKTCAIAQWISDVT547E55LOM5RMHRO6PHQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.316597522 (525h45m29.187680627s) [noise=true , address=145.79.227.34:4929 , piece_id=R6AMZGCVXGSVOTCHXP4JTGYRXUQCF5CAGKIFIQ4PW77F65HP77UA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.316807552 (525h45m29.187890587s) [noise=true , address=145.79.227.17:4998 , piece_id=7GPON22DFOGVWIFAFSKV5JCWRUGVYAWGSKPHOXU7ACH6AG3TEUJA, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:29.317160294 (525h45m29.188243329s) [noise=true , address=77.90.31.12:4613 , piece_id=UEKHT3CPCVS5FKTHTXI7C7OPLJCDSWWJEVZBKKCGXOVJQUSUOYTQ, seg_pos={0 0}, upload=461177]: Uploading piece... Dec 27 13:14:30.687966688 (525h45m30.559049733s) [upload=461177]: Next returned nil segment Dec 27 13:14:30.687980184 (525h45m30.559063219s) [upload=461177]: Waiting for error group managing segments... Dec 27 13:14:31.040556219 (525h45m30.911639264s) [noise=true , address=193.149.164.143:4701 , piece_id=WYPVV5ND6727W3RUHKVEBW7TY2SLK44OHGIWKNQS3Z7VRA7R6BHA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.181412965 (525h45m31.05249601s) [noise=true , address=145.79.227.48:4311 , piece_id=JPY7E33FC6AWCECY6IQEXPMRUSCR6EDJXRFARMU7ZWXS5D7RRVUQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.185708925 (525h45m31.056791961s) [noise=true , address=145.79.227.40:4062 , piece_id=PN5ZDTWCSQEQXCRG7Q2KWJ35JA26CIVQCEXIPYH7MSEFCW55JAXA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.220843017 (525h45m31.091926062s) [noise=true , address=145.79.227.17:4998 , piece_id=7GPON22DFOGVWIFAFSKV5JCWRUGVYAWGSKPHOXU7ACH6AG3TEUJA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.303221950 (525h45m31.174304995s) [noise=true , address=145.79.227.52:4478 , piece_id=TPSQDF6VYSA7UY4GOJBXN5HX7NGMHKXOS35ICXIUVQCDZYILISBA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.396094786 (525h45m31.267177831s) [noise=true , address=77.90.31.12:4613 , piece_id=UEKHT3CPCVS5FKTHTXI7C7OPLJCDSWWJEVZBKKCGXOVJQUSUOYTQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.467160147 (525h45m31.338243192s) [noise=true , address=77.90.31.14:4687 , piece_id=2NBPJXVK52L2M7HNYEH3V5XMA5TJ3O6C27PON23W5QLXYU3S6PAA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.494978408 (525h45m31.366061453s) [noise=true , address=77.90.31.19:4874 , piece_id=SXRHPQ6ZKSYS7XMDFM4NM444R3UD77XDCZCLDJGYNVXMC5ZSYWAQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.550392295 (525h45m31.42147534s) [noise=true , address=45.91.116.4:4964 , piece_id=PS273B6BPO4O6J6G4LQLCVRAOKP67PKW4NZRR5FPN5RPCB2MEQQQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.731063575 (525h45m31.60214661s) [noise=true , address=45.91.116.5:4329 , piece_id=4IOSYSIGU2FGHLMUUS5CTYAK3IVKC5UKEPFKTOOZJLUCHU46QVOQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:31.811147498 (525h45m31.682230543s) [noise=true , address=77.90.31.15:4712 , piece_id=PGSHIFGJWOYKM6E7PWUDTUZP6S32PETNZLD73GL3ZMIZP4REYQMA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:32.070197935 (525h45m31.94128098s) [noise=true , address=145.79.227.44:4225 , piece_id=45QAKVCQ5FPOLE5DRCG2WSFIOYO7C7NH4HTYHSQILOUFM7EFXO2Q, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:32.081701990 (525h45m31.952785025s) [noise=true , address=145.79.227.30:4768 , piece_id=DQ23KUHNR6GG5RA44UOSKTCAIAQWISDVT547E55LOM5RMHRO6PHQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:32.087629396 (525h45m31.958712432s) [noise=true , address=77.90.31.21:4894 , piece_id=GR5PRAIVWGPPSNDO2IKB7VN65UBASLHZELBLHORDGYOQA4LWZRSA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: <nil> Dec 27 13:14:32.087648273 (525h45m31.958731318s) [seg_pos={0 0}, upload=461177]: Segment reached optimal threshold of 14 pieces. Dec 27 13:14:32.087908097 (525h45m31.958991142s) [noise=true , address=64.49.13.169:4710 , piece_id=TIN6WHF6JQRHYT5BYZISPGW6D2QUVYZ4RXVJHBPVZEHJQXFXY2FQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:41926->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:41926->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41926->64.49.13.169:4710: use of closed network connection Dec 27 13:14:32.087914629 (525h45m31.958997664s) [noise=true , address=64.49.13.169:4710 , piece_id=TIN6WHF6JQRHYT5BYZISPGW6D2QUVYZ4RXVJHBPVZEHJQXFXY2FQ, seg_pos={0 0}, upload=461177]: Long tail cancellation due to optimal threshold Dec 27 13:14:32.087964986 (525h45m31.959048021s) [noise=true , address=77.90.31.16:4759 , piece_id=GG44CNIRWLO5NTIZPNEEQY522WZQBRKZYRKW6QMICCNKF6FPL45A, seg_pos={0 0}, upload=461177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:32.087980906 (525h45m31.959063941s) [noise=true , address=77.90.31.16:4759 , piece_id=GG44CNIRWLO5NTIZPNEEQY522WZQBRKZYRKW6QMICCNKF6FPL45A, seg_pos={0 0}, upload=461177]: Long tail cancellation due to optimal threshold Dec 27 13:14:32.088048234 (525h45m31.959131279s) [noise=true , address=45.91.116.10:4089 , piece_id=63NTMXR7WBHBQ52HJUCB2SE5NBCPSZVBECHIKD3LPYFTM7KA3ZOQ, seg_pos={0 0}, upload=461177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UGEGmuAkzSPKBqfvt54v431PqBuq8nq7f3KeLQ8p4opnDRqTK): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:32.088053634 (525h45m31.959136669s) [noise=true , address=45.91.116.10:4089 , piece_id=63NTMXR7WBHBQ52HJUCB2SE5NBCPSZVBECHIKD3LPYFTM7KA3ZOQ, seg_pos={0 0}, upload=461177]: Long tail cancellation due to optimal threshold Dec 27 13:14:32.088203389 (525h45m31.959286434s) [noise=true , address=145.79.227.34:4929 , piece_id=R6AMZGCVXGSVOTCHXP4JTGYRXUQCF5CAGKIFIQ4PW77F65HP77UA, seg_pos={0 0}, upload=461177]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:32.088208870 (525h45m31.959291905s) [noise=true , address=145.79.227.34:4929 , piece_id=R6AMZGCVXGSVOTCHXP4JTGYRXUQCF5CAGKIFIQ4PW77F65HP77UA, seg_pos={0 0}, upload=461177]: Long tail cancellation due to optimal threshold Dec 27 13:14:32.088227615 (525h45m31.95931065s) [seg_pos={0 0}, upload=461177]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:14:32.088294693 (525h45m31.959377728s) [upload=461177]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:14:32.088299031 (525h45m31.959382066s) [upload=461177]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:14:32.389880444 (525h45m32.260963489s) [upload=461177]: Done uploading segments... Dec 27 13:14:32.389899390 (525h45m32.260982426s) [upload=461177]: Upload finished. err: <nil> Duration: 1m0.002270099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [........X.X......X..............X..X.X...X.XX..XX.X.......XX] duration: 5.001035333s trace: https://jaeger.dev.storj.io/trace/4526d2c427d3e740 FAIL (2025-12-27 13:15:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001077383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X.X..XX.X......X..X..XXX...........XX..X...XX......X..X.] duration: 5.920612156s trace: https://jaeger.dev.storj.io/trace/66a858bc169820a3 FAIL (2025-12-27 13:14:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 13:14:00.930597344 (525h45m0.80168039s) [upload=461170]: Starting upload Dec 27 13:14:00.930690642 (525h45m0.801773718s) [upload=461170]: Done starting upload Dec 27 13:14:00.930803036 (525h45m0.801886081s) [upload=461170]: Uploading segments... Dec 27 13:14:00.930978100 (525h45m0.802061145s) [upload=461170]: Got next segment. Inline: false Dec 27 13:14:00.931020871 (525h45m0.802103906s) [upload=461170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:14:00.931027663 (525h45m0.802110709s) [upload=461170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:14:01.176830284 (525h45m1.047913319s) [seg_pos={0 0}, upload=461170]: Begin upload segment... Dec 27 13:14:01.176905156 (525h45m1.047988191s) [seg_pos={0 0}, upload=461170]: Done begin upload segment. Dec 27 13:14:01.177015857 (525h45m1.048098902s) [noise=true , address=77.90.31.10:4517 , piece_id=XBIASFXOPMDKDK7AE3HQRHXVDNJGG5A6EEVNU7MCJ7MJDPHQ2NTQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.177105558 (525h45m1.048188603s) [noise=true , address=145.79.227.45:4231 , piece_id=QPZWGHITJ4TH6AAOKEZCI3UFGNERNHQSEDFQEDDQNSMLAZNTUXZA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.177668801 (525h45m1.048751846s) [noise=true , address=145.79.227.37:4566 , piece_id=26XNZ4FAC7MT2WYBQSGWOZT3AQ5GFDNSHWOXRJULY4UYREYADS3A, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.178230680 (525h45m1.049313715s) [noise=true , address=145.79.227.36:4518 , piece_id=53D2BGXYKZPOH77NGYZY64IGQ3JWQJEODCIUYNHP7FZ52EFL7FXA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.179606671 (525h45m1.050689716s) [noise=true , address=77.90.31.14:4661 , piece_id=YIKZB3QDUYR2Q5VRPEX3BSK2IVCX4UQGYEXQUG4MQOEOA2XKNTFQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.180250296 (525h45m1.051333341s) [noise=true , address=77.90.31.11:4554 , piece_id=QDGSXF6Y6KN7HF5UKEFNWBMB7SQ6TMGRWMANLCU5EZ7BMAYLM4DQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.181620325 (525h45m1.052703371s) [noise=true , address=77.90.31.12:4590 , piece_id=HG6DWEFAOLXYJ36YBBJQLM546AZ3IEMZX3AVPZABFTF4KJC7BMRA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.181920326 (525h45m1.053003371s) [noise=true , address=45.91.116.12:4226 , piece_id=XPTZ55NVVT67SMQ24CDVVNKVCT774QEUNCGBDAIGVEYMIRL2SXRQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.183146732 (525h45m1.054229777s) [noise=true , address=145.79.227.42:4123 , piece_id=G4K3B5CGIN4EF5DDAXAQBNTA74QRX76BMG3VOC4ONBM72FSWJKZA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.183911218 (525h45m1.054994253s) [noise=true , address=77.90.31.21:4880 , piece_id=DL3S24B5QP2JSML25CGUAOXNZP3KJWYAV7JY3IDK67PRQ6O727GQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.185980387 (525h45m1.057063443s) [noise=true , address=77.90.31.19:4841 , piece_id=PQPFCVGQ42FTXXOV7MSOREIZK6QPFAUCYDPVAIL5HKXJDLL52SWQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.188163595 (525h45m1.05924665s) [noise=true , address=145.79.227.39:4015 , piece_id=IMVEMCHUQWHJPR4ACRB2Z7IDZXM4ZGW4RWYSC66TXTCRI3YLEQSA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.189639545 (525h45m1.06072259s) [noise=true , address=145.79.227.11:4732 , piece_id=NNSTTBHYAQJPPBXECUZE4T24JQBFZSFHGWZ3ERRR5PAJ2I3SUNZA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.191979010 (525h45m1.063062055s) [noise=true , address=145.79.227.2:4114 , piece_id=Q2MWW6QINH2UIU47NLBZOECNECDLQD5ZMT7JSHJPFB733S6F73PA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.194386303 (525h45m1.065469348s) [noise=true , address=145.79.227.46:4603 , piece_id=TZ7FWPRWMQRIPAOBD47SFVDSMHOET5KFZ63UAY6XEIH25BWWNRKA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.195559157 (525h45m1.066642202s) [noise=true , address=145.79.227.51:4410 , piece_id=SEZMCOVJXDXC5IWB26N6TMHACJX467SOWQSNPZ2F3WVATQUISWGQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:02.799598243 (525h45m2.670681299s) [upload=461170]: Next returned nil segment Dec 27 13:14:02.799619344 (525h45m2.670702389s) [upload=461170]: Waiting for error group managing segments... Dec 27 13:14:03.185957448 (525h45m3.057040493s) [noise=true , address=145.79.227.42:4123 , piece_id=G4K3B5CGIN4EF5DDAXAQBNTA74QRX76BMG3VOC4ONBM72FSWJKZA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.188334494 (525h45m3.059417539s) [noise=true , address=145.79.227.2:4114 , piece_id=Q2MWW6QINH2UIU47NLBZOECNECDLQD5ZMT7JSHJPFB733S6F73PA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.189580988 (525h45m3.060664033s) [noise=true , address=145.79.227.36:4518 , piece_id=53D2BGXYKZPOH77NGYZY64IGQ3JWQJEODCIUYNHP7FZ52EFL7FXA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.196858015 (525h45m3.06794106s) [noise=true , address=145.79.227.39:4015 , piece_id=IMVEMCHUQWHJPR4ACRB2Z7IDZXM4ZGW4RWYSC66TXTCRI3YLEQSA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.199856634 (525h45m3.070939679s) [noise=true , address=145.79.227.37:4566 , piece_id=26XNZ4FAC7MT2WYBQSGWOZT3AQ5GFDNSHWOXRJULY4UYREYADS3A, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.210819920 (525h45m3.081902976s) [noise=true , address=145.79.227.46:4603 , piece_id=TZ7FWPRWMQRIPAOBD47SFVDSMHOET5KFZ63UAY6XEIH25BWWNRKA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.210854757 (525h45m3.081937802s) [noise=true , address=145.79.227.45:4231 , piece_id=QPZWGHITJ4TH6AAOKEZCI3UFGNERNHQSEDFQEDDQNSMLAZNTUXZA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.217484562 (525h45m3.088567607s) [noise=true , address=145.79.227.51:4410 , piece_id=SEZMCOVJXDXC5IWB26N6TMHACJX467SOWQSNPZ2F3WVATQUISWGQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.219463480 (525h45m3.090546525s) [noise=true , address=145.79.227.11:4732 , piece_id=NNSTTBHYAQJPPBXECUZE4T24JQBFZSFHGWZ3ERRR5PAJ2I3SUNZA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.395263393 (525h45m3.266346468s) [noise=true , address=77.90.31.11:4554 , piece_id=QDGSXF6Y6KN7HF5UKEFNWBMB7SQ6TMGRWMANLCU5EZ7BMAYLM4DQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.433922256 (525h45m3.305005301s) [noise=true , address=77.90.31.21:4880 , piece_id=DL3S24B5QP2JSML25CGUAOXNZP3KJWYAV7JY3IDK67PRQ6O727GQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.442870405 (525h45m3.31395345s) [noise=true , address=77.90.31.12:4590 , piece_id=HG6DWEFAOLXYJ36YBBJQLM546AZ3IEMZX3AVPZABFTF4KJC7BMRA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.451919566 (525h45m3.323002611s) [noise=true , address=77.90.31.19:4841 , piece_id=PQPFCVGQ42FTXXOV7MSOREIZK6QPFAUCYDPVAIL5HKXJDLL52SWQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.451970804 (525h45m3.323053849s) [seg_pos={0 0}, upload=461170]: Segment reached optimal threshold of 13 pieces. Dec 27 13:14:03.452400842 (525h45m3.323483887s) [noise=true , address=77.90.31.10:4517 , piece_id=XBIASFXOPMDKDK7AE3HQRHXVDNJGG5A6EEVNU7MCJ7MJDPHQ2NTQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:03.452425550 (525h45m3.323508605s) [noise=true , address=77.90.31.10:4517 , piece_id=XBIASFXOPMDKDK7AE3HQRHXVDNJGG5A6EEVNU7MCJ7MJDPHQ2NTQ, seg_pos={0 0}, upload=461170]: Long tail cancellation due to optimal threshold Dec 27 13:14:03.452608748 (525h45m3.323691793s) [noise=true , address=77.90.31.14:4661 , piece_id=YIKZB3QDUYR2Q5VRPEX3BSK2IVCX4UQGYEXQUG4MQOEOA2XKNTFQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:03.452618477 (525h45m3.323701522s) [noise=true , address=77.90.31.14:4661 , piece_id=YIKZB3QDUYR2Q5VRPEX3BSK2IVCX4UQGYEXQUG4MQOEOA2XKNTFQ, seg_pos={0 0}, upload=461170]: Long tail cancellation due to optimal threshold Dec 27 13:14:03.452714790 (525h45m3.323797825s) [noise=true , address=45.91.116.12:4226 , piece_id=XPTZ55NVVT67SMQ24CDVVNKVCT774QEUNCGBDAIGVEYMIRL2SXRQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:03.452724558 (525h45m3.323807594s) [noise=true , address=45.91.116.12:4226 , piece_id=XPTZ55NVVT67SMQ24CDVVNKVCT774QEUNCGBDAIGVEYMIRL2SXRQ, seg_pos={0 0}, upload=461170]: Long tail cancellation due to optimal threshold Dec 27 13:14:03.452788731 (525h45m3.323871766s) [seg_pos={0 0}, upload=461170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:14:03.452885345 (525h45m3.32396839s) [upload=461170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:14:03.452893611 (525h45m3.323976646s) [upload=461170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:14:03.747125409 (525h45m3.618208454s) [upload=461170]: Done uploading segments... Dec 27 13:14:03.747154284 (525h45m3.618237329s) [upload=461170]: Upload finished. err: <nil> Duration: 5.920633966s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X.X..XX.X......X..X..XXX...........XX..X...XX......X..X] duration: 30.003348919s trace: https://jaeger.dev.storj.io/trace/66a858bc169820a3 FAIL (2025-12-27 13:14:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 13:14:00.930597344 (525h45m0.80168039s) [upload=461170]: Starting upload Dec 27 13:14:00.930690642 (525h45m0.801773718s) [upload=461170]: Done starting upload Dec 27 13:14:00.930803036 (525h45m0.801886081s) [upload=461170]: Uploading segments... Dec 27 13:14:00.930978100 (525h45m0.802061145s) [upload=461170]: Got next segment. Inline: false Dec 27 13:14:00.931020871 (525h45m0.802103906s) [upload=461170]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:14:00.931027663 (525h45m0.802110709s) [upload=461170]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:14:01.176830284 (525h45m1.047913319s) [seg_pos={0 0}, upload=461170]: Begin upload segment... Dec 27 13:14:01.176905156 (525h45m1.047988191s) [seg_pos={0 0}, upload=461170]: Done begin upload segment. Dec 27 13:14:01.177015857 (525h45m1.048098902s) [noise=true , address=77.90.31.10:4517 , piece_id=XBIASFXOPMDKDK7AE3HQRHXVDNJGG5A6EEVNU7MCJ7MJDPHQ2NTQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.177105558 (525h45m1.048188603s) [noise=true , address=145.79.227.45:4231 , piece_id=QPZWGHITJ4TH6AAOKEZCI3UFGNERNHQSEDFQEDDQNSMLAZNTUXZA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.177668801 (525h45m1.048751846s) [noise=true , address=145.79.227.37:4566 , piece_id=26XNZ4FAC7MT2WYBQSGWOZT3AQ5GFDNSHWOXRJULY4UYREYADS3A, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.178230680 (525h45m1.049313715s) [noise=true , address=145.79.227.36:4518 , piece_id=53D2BGXYKZPOH77NGYZY64IGQ3JWQJEODCIUYNHP7FZ52EFL7FXA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.179606671 (525h45m1.050689716s) [noise=true , address=77.90.31.14:4661 , piece_id=YIKZB3QDUYR2Q5VRPEX3BSK2IVCX4UQGYEXQUG4MQOEOA2XKNTFQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.180250296 (525h45m1.051333341s) [noise=true , address=77.90.31.11:4554 , piece_id=QDGSXF6Y6KN7HF5UKEFNWBMB7SQ6TMGRWMANLCU5EZ7BMAYLM4DQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.181620325 (525h45m1.052703371s) [noise=true , address=77.90.31.12:4590 , piece_id=HG6DWEFAOLXYJ36YBBJQLM546AZ3IEMZX3AVPZABFTF4KJC7BMRA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.181920326 (525h45m1.053003371s) [noise=true , address=45.91.116.12:4226 , piece_id=XPTZ55NVVT67SMQ24CDVVNKVCT774QEUNCGBDAIGVEYMIRL2SXRQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.183146732 (525h45m1.054229777s) [noise=true , address=145.79.227.42:4123 , piece_id=G4K3B5CGIN4EF5DDAXAQBNTA74QRX76BMG3VOC4ONBM72FSWJKZA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.183911218 (525h45m1.054994253s) [noise=true , address=77.90.31.21:4880 , piece_id=DL3S24B5QP2JSML25CGUAOXNZP3KJWYAV7JY3IDK67PRQ6O727GQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.185980387 (525h45m1.057063443s) [noise=true , address=77.90.31.19:4841 , piece_id=PQPFCVGQ42FTXXOV7MSOREIZK6QPFAUCYDPVAIL5HKXJDLL52SWQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.188163595 (525h45m1.05924665s) [noise=true , address=145.79.227.39:4015 , piece_id=IMVEMCHUQWHJPR4ACRB2Z7IDZXM4ZGW4RWYSC66TXTCRI3YLEQSA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.189639545 (525h45m1.06072259s) [noise=true , address=145.79.227.11:4732 , piece_id=NNSTTBHYAQJPPBXECUZE4T24JQBFZSFHGWZ3ERRR5PAJ2I3SUNZA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.191979010 (525h45m1.063062055s) [noise=true , address=145.79.227.2:4114 , piece_id=Q2MWW6QINH2UIU47NLBZOECNECDLQD5ZMT7JSHJPFB733S6F73PA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.194386303 (525h45m1.065469348s) [noise=true , address=145.79.227.46:4603 , piece_id=TZ7FWPRWMQRIPAOBD47SFVDSMHOET5KFZ63UAY6XEIH25BWWNRKA, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:01.195559157 (525h45m1.066642202s) [noise=true , address=145.79.227.51:4410 , piece_id=SEZMCOVJXDXC5IWB26N6TMHACJX467SOWQSNPZ2F3WVATQUISWGQ, seg_pos={0 0}, upload=461170]: Uploading piece... Dec 27 13:14:02.799598243 (525h45m2.670681299s) [upload=461170]: Next returned nil segment Dec 27 13:14:02.799619344 (525h45m2.670702389s) [upload=461170]: Waiting for error group managing segments... Dec 27 13:14:03.185957448 (525h45m3.057040493s) [noise=true , address=145.79.227.42:4123 , piece_id=G4K3B5CGIN4EF5DDAXAQBNTA74QRX76BMG3VOC4ONBM72FSWJKZA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.188334494 (525h45m3.059417539s) [noise=true , address=145.79.227.2:4114 , piece_id=Q2MWW6QINH2UIU47NLBZOECNECDLQD5ZMT7JSHJPFB733S6F73PA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.189580988 (525h45m3.060664033s) [noise=true , address=145.79.227.36:4518 , piece_id=53D2BGXYKZPOH77NGYZY64IGQ3JWQJEODCIUYNHP7FZ52EFL7FXA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.196858015 (525h45m3.06794106s) [noise=true , address=145.79.227.39:4015 , piece_id=IMVEMCHUQWHJPR4ACRB2Z7IDZXM4ZGW4RWYSC66TXTCRI3YLEQSA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.199856634 (525h45m3.070939679s) [noise=true , address=145.79.227.37:4566 , piece_id=26XNZ4FAC7MT2WYBQSGWOZT3AQ5GFDNSHWOXRJULY4UYREYADS3A, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.210819920 (525h45m3.081902976s) [noise=true , address=145.79.227.46:4603 , piece_id=TZ7FWPRWMQRIPAOBD47SFVDSMHOET5KFZ63UAY6XEIH25BWWNRKA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.210854757 (525h45m3.081937802s) [noise=true , address=145.79.227.45:4231 , piece_id=QPZWGHITJ4TH6AAOKEZCI3UFGNERNHQSEDFQEDDQNSMLAZNTUXZA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.217484562 (525h45m3.088567607s) [noise=true , address=145.79.227.51:4410 , piece_id=SEZMCOVJXDXC5IWB26N6TMHACJX467SOWQSNPZ2F3WVATQUISWGQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.219463480 (525h45m3.090546525s) [noise=true , address=145.79.227.11:4732 , piece_id=NNSTTBHYAQJPPBXECUZE4T24JQBFZSFHGWZ3ERRR5PAJ2I3SUNZA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.395263393 (525h45m3.266346468s) [noise=true , address=77.90.31.11:4554 , piece_id=QDGSXF6Y6KN7HF5UKEFNWBMB7SQ6TMGRWMANLCU5EZ7BMAYLM4DQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.433922256 (525h45m3.305005301s) [noise=true , address=77.90.31.21:4880 , piece_id=DL3S24B5QP2JSML25CGUAOXNZP3KJWYAV7JY3IDK67PRQ6O727GQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.442870405 (525h45m3.31395345s) [noise=true , address=77.90.31.12:4590 , piece_id=HG6DWEFAOLXYJ36YBBJQLM546AZ3IEMZX3AVPZABFTF4KJC7BMRA, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.451919566 (525h45m3.323002611s) [noise=true , address=77.90.31.19:4841 , piece_id=PQPFCVGQ42FTXXOV7MSOREIZK6QPFAUCYDPVAIL5HKXJDLL52SWQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: <nil> Dec 27 13:14:03.451970804 (525h45m3.323053849s) [seg_pos={0 0}, upload=461170]: Segment reached optimal threshold of 13 pieces. Dec 27 13:14:03.452400842 (525h45m3.323483887s) [noise=true , address=77.90.31.10:4517 , piece_id=XBIASFXOPMDKDK7AE3HQRHXVDNJGG5A6EEVNU7MCJ7MJDPHQ2NTQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:03.452425550 (525h45m3.323508605s) [noise=true , address=77.90.31.10:4517 , piece_id=XBIASFXOPMDKDK7AE3HQRHXVDNJGG5A6EEVNU7MCJ7MJDPHQ2NTQ, seg_pos={0 0}, upload=461170]: Long tail cancellation due to optimal threshold Dec 27 13:14:03.452608748 (525h45m3.323691793s) [noise=true , address=77.90.31.14:4661 , piece_id=YIKZB3QDUYR2Q5VRPEX3BSK2IVCX4UQGYEXQUG4MQOEOA2XKNTFQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:03.452618477 (525h45m3.323701522s) [noise=true , address=77.90.31.14:4661 , piece_id=YIKZB3QDUYR2Q5VRPEX3BSK2IVCX4UQGYEXQUG4MQOEOA2XKNTFQ, seg_pos={0 0}, upload=461170]: Long tail cancellation due to optimal threshold Dec 27 13:14:03.452714790 (525h45m3.323797825s) [noise=true , address=45.91.116.12:4226 , piece_id=XPTZ55NVVT67SMQ24CDVVNKVCT774QEUNCGBDAIGVEYMIRL2SXRQ, seg_pos={0 0}, upload=461170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 13:14:03.452724558 (525h45m3.323807594s) [noise=true , address=45.91.116.12:4226 , piece_id=XPTZ55NVVT67SMQ24CDVVNKVCT774QEUNCGBDAIGVEYMIRL2SXRQ, seg_pos={0 0}, upload=461170]: Long tail cancellation due to optimal threshold Dec 27 13:14:03.452788731 (525h45m3.323871766s) [seg_pos={0 0}, upload=461170]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:14:03.452885345 (525h45m3.32396839s) [upload=461170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:14:03.452893611 (525h45m3.323976646s) [upload=461170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:14:03.747125409 (525h45m3.618208454s) [upload=461170]: Done uploading segments... Dec 27 13:14:03.747154284 (525h45m3.618237329s) [upload=461170]: Upload finished. err: <nil> Duration: 30.003389136s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.........X.X......X..............X..X.X...X.XX..XX.X.......X] duration: 5.000910476s trace: https://jaeger.dev.storj.io/trace/3465355956561979 FAIL (2025-12-27 13:14:23 UTC) orbiter warning: context deadline exceeded Duration: 5.00093375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.............................X...] duration: 52.676992064s trace: https://jaeger.dev.storj.io/trace/15e7e71b1e4a765b FAIL (2025-12-27 13:14:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: TPMTEEA74KLTJ5V6YIMXCNHVM7CMXNAPI5JDH53LM3MBII42BGEA) context deadline exceeded; piecestore: (Node ID: 1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, Piece ID: TZMEUIWTKW7STBEA6VFITVALLVXHIED6C7JVPUYL73CSC2AG5YSA) context deadline exceeded; piecestore: (Node ID: 12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, Piece ID: QOZBFTVMOQ3CAF7BLJVABEIVKIKGXXW6QNLYN27CGTMGG5K2N2BQ) context deadline exceeded ======LOGS====== oise=true , address=77.90.31.13:4651 , piece_id=YWKS2JTDS5QYYPTEEOF54N7LKZGCA6UXQT2DNUA5KRZUXSJ73CZQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.709860762 (525h41m45.580943797s) [noise=true , address=64.49.13.170:4724 , piece_id=FRNFGNNL57GYHDY2EB2DXRQEHUZQTKYYIS3475N7R54H5HATCXVQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.710306931 (525h41m45.581389976s) [noise=true , address=145.79.227.34:4911 , piece_id=M3ODVIPMMESPK5PJTT2N7TX2N6WIZAMKH4U7HMEOVHDPQMWIS5ZA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.710677497 (525h41m45.581760532s) [noise=true , address=77.90.31.12:4624 , piece_id=XUYFLD7JCKYE2LQ66PVOOSRJV4NUZI3VQN6CMG6AUQJETYYN5BCA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.711052121 (525h41m45.582135156s) [noise=true , address=77.90.31.19:4863 , piece_id=6ZLPLFQDENXYKH5BPW3IAMJY7QIDI2SUIRARCBYKNDMOE5BLGTMA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.711430001 (525h41m45.582513036s) [noise=true , address=145.79.227.16:4932 , piece_id=425NEIP6DT3OBKARRSWY76MYMSAZ6VYUWDEY2TKQDJM66F6DQ2YQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.711825103 (525h41m45.582908138s) [noise=true , address=77.90.31.18:4833 , piece_id=TPMTEEA74KLTJ5V6YIMXCNHVM7CMXNAPI5JDH53LM3MBII42BGEA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.712248579 (525h41m45.583331624s) [noise=true , address=145.79.227.40:4078 , piece_id=5JTMTHVDQIR5XO7EBLLBQVVTYB47SQSEIL6F6Y7AZWNH2PKPOYYQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.712670562 (525h41m45.583753607s) [noise=true , address=77.90.31.11:4565 , piece_id=N56S54JPLASZYOGS6VX6YK2ED6GWJDIKCI6TZPGIEFSA4LXEAZJA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.713324337 (525h41m45.584407372s) [noise=true , address=64.49.13.169:4707 , piece_id=CMVRTRWEMPOFKBA6DJ7TOVSCM56IOGVDNKP327EGO2CREAYWF6WA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.713941212 (525h41m45.585024247s) [noise=true , address=145.79.227.43:4162 , piece_id=5D3REI4QA47RUUN3IMIOW6CXFZZAZRYLKVAZCL6YR4MPG7UV4KPQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.715054432 (525h41m45.586137467s) [noise=true , address=77.90.31.15:4718 , piece_id=IHG7SIDROBAPTLWNBEXSIZYG5L4LHI7N4CEPY3APQZMEQJFRACRQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.715571597 (525h41m45.586654632s) [noise=true , address=45.91.116.7:4020 , piece_id=TZMEUIWTKW7STBEA6VFITVALLVXHIED6C7JVPUYL73CSC2AG5YSA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.716503260 (525h41m45.587586305s) [noise=true , address=77.90.31.21:4907 , piece_id=KV4WOT4TC5QJEUEXWGZ6TPHMYALHLB3ZEOW35RZVFTCMGJVWH6ZA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.717147277 (525h41m45.588230312s) [noise=true , address=145.79.227.30:4796 , piece_id=ISVTXMY2AMJI3RDRURII7CX4R7PVPCW7ES7TZFMEV4TXPUU4RYVA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.718560367 (525h41m45.589643402s) [noise=true , address=77.90.31.17:4792 , piece_id=QOZBFTVMOQ3CAF7BLJVABEIVKIKGXXW6QNLYN27CGTMGG5K2N2BQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:47.060058932 (525h41m46.931141967s) [upload=461126]: Next returned nil segment Dec 27 13:10:47.060073020 (525h41m46.931156065s) [upload=461126]: Waiting for error group managing segments... Dec 27 13:10:47.409551807 (525h41m47.280634842s) [noise=true , address=64.49.13.170:4724 , piece_id=FRNFGNNL57GYHDY2EB2DXRQEHUZQTKYYIS3475N7R54H5HATCXVQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.599519939 (525h41m47.470602974s) [noise=true , address=145.79.227.16:4932 , piece_id=425NEIP6DT3OBKARRSWY76MYMSAZ6VYUWDEY2TKQDJM66F6DQ2YQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.605459099 (525h41m47.476542144s) [noise=true , address=145.79.227.43:4162 , piece_id=5D3REI4QA47RUUN3IMIOW6CXFZZAZRYLKVAZCL6YR4MPG7UV4KPQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.707001222 (525h41m47.578084257s) [noise=true , address=145.79.227.34:4911 , piece_id=M3ODVIPMMESPK5PJTT2N7TX2N6WIZAMKH4U7HMEOVHDPQMWIS5ZA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.820931805 (525h41m47.69201485s) [noise=true , address=77.90.31.18:4833 , piece_id=TPMTEEA74KLTJ5V6YIMXCNHVM7CMXNAPI5JDH53LM3MBII42BGEA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.824299398 (525h41m47.695382433s) [noise=true , address=45.91.116.7:4020 , piece_id=TZMEUIWTKW7STBEA6VFITVALLVXHIED6C7JVPUYL73CSC2AG5YSA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.825288943 (525h41m47.696371978s) [noise=true , address=77.90.31.17:4792 , piece_id=QOZBFTVMOQ3CAF7BLJVABEIVKIKGXXW6QNLYN27CGTMGG5K2N2BQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.852715927 (525h41m47.723798972s) [noise=true , address=77.90.31.19:4863 , piece_id=6ZLPLFQDENXYKH5BPW3IAMJY7QIDI2SUIRARCBYKNDMOE5BLGTMA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.008917121 (525h41m47.880000166s) [noise=true , address=77.90.31.21:4907 , piece_id=KV4WOT4TC5QJEUEXWGZ6TPHMYALHLB3ZEOW35RZVFTCMGJVWH6ZA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.038473493 (525h41m47.909556538s) [noise=true , address=77.90.31.12:4624 , piece_id=XUYFLD7JCKYE2LQ66PVOOSRJV4NUZI3VQN6CMG6AUQJETYYN5BCA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.505751901 (525h41m48.376834936s) [noise=true , address=145.79.227.44:4228 , piece_id=TVURXWI4IKDIXJ34HYZ5D4B4FGR6U4FUCZXACIV2GNGDPO4SAOLQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.521156504 (525h41m48.392239539s) [noise=true , address=145.79.227.30:4796 , piece_id=ISVTXMY2AMJI3RDRURII7CX4R7PVPCW7ES7TZFMEV4TXPUU4RYVA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.545623223 (525h41m48.416706268s) [noise=true , address=77.90.31.16:4745 , piece_id=7K2OUZKFBFNXQMDWDNJHYAA3LIHCIA2EL7FLKMVKOL744V4JN6GQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.552502362 (525h41m48.423585397s) [noise=true , address=145.79.227.40:4078 , piece_id=5JTMTHVDQIR5XO7EBLLBQVVTYB47SQSEIL6F6Y7AZWNH2PKPOYYQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.552521197 (525h41m48.423604232s) [seg_pos={0 0}, upload=461126]: Segment reached optimal threshold of 14 pieces. Dec 27 13:10:48.552693505 (525h41m48.423776551s) [noise=true , address=64.49.13.169:4707 , piece_id=CMVRTRWEMPOFKBA6DJ7TOVSCM56IOGVDNKP327EGO2CREAYWF6WA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:43628->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:43628->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43628->64.49.13.169:4707: use of closed network connection Dec 27 13:10:48.552701241 (525h41m48.423784276s) [noise=true , address=64.49.13.169:4707 , piece_id=CMVRTRWEMPOFKBA6DJ7TOVSCM56IOGVDNKP327EGO2CREAYWF6WA, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553068299 (525h41m48.424151344s) [noise=true , address=77.90.31.15:4718 , piece_id=IHG7SIDROBAPTLWNBEXSIZYG5L4LHI7N4CEPY3APQZMEQJFRACRQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: expected piece hash; context canceled; EOF Dec 27 13:10:48.553078509 (525h41m48.424161554s) [noise=true , address=77.90.31.15:4718 , piece_id=IHG7SIDROBAPTLWNBEXSIZYG5L4LHI7N4CEPY3APQZMEQJFRACRQ, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553264022 (525h41m48.424347068s) [noise=true , address=77.90.31.13:4651 , piece_id=YWKS2JTDS5QYYPTEEOF54N7LKZGCA6UXQT2DNUA5KRZUXSJ73CZQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 27 13:10:48.553272158 (525h41m48.424355193s) [noise=true , address=77.90.31.13:4651 , piece_id=YWKS2JTDS5QYYPTEEOF54N7LKZGCA6UXQT2DNUA5KRZUXSJ73CZQ, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553429377 (525h41m48.424512412s) [noise=true , address=77.90.31.11:4565 , piece_id=N56S54JPLASZYOGS6VX6YK2ED6GWJDIKCI6TZPGIEFSA4LXEAZJA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: expected piece hash; context canceled; EOF Dec 27 13:10:48.553437763 (525h41m48.424520798s) [noise=true , address=77.90.31.11:4565 , piece_id=N56S54JPLASZYOGS6VX6YK2ED6GWJDIKCI6TZPGIEFSA4LXEAZJA, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553456298 (525h41m48.424539343s) [seg_pos={0 0}, upload=461126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:10:48.553546450 (525h41m48.424629495s) [upload=461126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:10:48.553553333 (525h41m48.424636388s) [upload=461126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:10:48.860753135 (525h41m48.73183618s) [upload=461126]: Done uploading segments... Dec 27 13:10:48.860772321 (525h41m48.731855356s) [upload=461126]: Upload finished. err: <nil> Duration: 52.677011933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.............................X..] duration: 42.247750864s trace: https://jaeger.dev.storj.io/trace/15e7e71b1e4a765b FAIL (2025-12-27 13:13:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: TPMTEEA74KLTJ5V6YIMXCNHVM7CMXNAPI5JDH53LM3MBII42BGEA) context deadline exceeded; piecestore: (Node ID: 1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, Piece ID: TZMEUIWTKW7STBEA6VFITVALLVXHIED6C7JVPUYL73CSC2AG5YSA) context deadline exceeded; piecestore: (Node ID: 12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, Piece ID: QOZBFTVMOQ3CAF7BLJVABEIVKIKGXXW6QNLYN27CGTMGG5K2N2BQ) context deadline exceeded ======LOGS====== oise=true , address=77.90.31.13:4651 , piece_id=YWKS2JTDS5QYYPTEEOF54N7LKZGCA6UXQT2DNUA5KRZUXSJ73CZQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.709860762 (525h41m45.580943797s) [noise=true , address=64.49.13.170:4724 , piece_id=FRNFGNNL57GYHDY2EB2DXRQEHUZQTKYYIS3475N7R54H5HATCXVQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.710306931 (525h41m45.581389976s) [noise=true , address=145.79.227.34:4911 , piece_id=M3ODVIPMMESPK5PJTT2N7TX2N6WIZAMKH4U7HMEOVHDPQMWIS5ZA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.710677497 (525h41m45.581760532s) [noise=true , address=77.90.31.12:4624 , piece_id=XUYFLD7JCKYE2LQ66PVOOSRJV4NUZI3VQN6CMG6AUQJETYYN5BCA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.711052121 (525h41m45.582135156s) [noise=true , address=77.90.31.19:4863 , piece_id=6ZLPLFQDENXYKH5BPW3IAMJY7QIDI2SUIRARCBYKNDMOE5BLGTMA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.711430001 (525h41m45.582513036s) [noise=true , address=145.79.227.16:4932 , piece_id=425NEIP6DT3OBKARRSWY76MYMSAZ6VYUWDEY2TKQDJM66F6DQ2YQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.711825103 (525h41m45.582908138s) [noise=true , address=77.90.31.18:4833 , piece_id=TPMTEEA74KLTJ5V6YIMXCNHVM7CMXNAPI5JDH53LM3MBII42BGEA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.712248579 (525h41m45.583331624s) [noise=true , address=145.79.227.40:4078 , piece_id=5JTMTHVDQIR5XO7EBLLBQVVTYB47SQSEIL6F6Y7AZWNH2PKPOYYQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.712670562 (525h41m45.583753607s) [noise=true , address=77.90.31.11:4565 , piece_id=N56S54JPLASZYOGS6VX6YK2ED6GWJDIKCI6TZPGIEFSA4LXEAZJA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.713324337 (525h41m45.584407372s) [noise=true , address=64.49.13.169:4707 , piece_id=CMVRTRWEMPOFKBA6DJ7TOVSCM56IOGVDNKP327EGO2CREAYWF6WA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.713941212 (525h41m45.585024247s) [noise=true , address=145.79.227.43:4162 , piece_id=5D3REI4QA47RUUN3IMIOW6CXFZZAZRYLKVAZCL6YR4MPG7UV4KPQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.715054432 (525h41m45.586137467s) [noise=true , address=77.90.31.15:4718 , piece_id=IHG7SIDROBAPTLWNBEXSIZYG5L4LHI7N4CEPY3APQZMEQJFRACRQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.715571597 (525h41m45.586654632s) [noise=true , address=45.91.116.7:4020 , piece_id=TZMEUIWTKW7STBEA6VFITVALLVXHIED6C7JVPUYL73CSC2AG5YSA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.716503260 (525h41m45.587586305s) [noise=true , address=77.90.31.21:4907 , piece_id=KV4WOT4TC5QJEUEXWGZ6TPHMYALHLB3ZEOW35RZVFTCMGJVWH6ZA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.717147277 (525h41m45.588230312s) [noise=true , address=145.79.227.30:4796 , piece_id=ISVTXMY2AMJI3RDRURII7CX4R7PVPCW7ES7TZFMEV4TXPUU4RYVA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.718560367 (525h41m45.589643402s) [noise=true , address=77.90.31.17:4792 , piece_id=QOZBFTVMOQ3CAF7BLJVABEIVKIKGXXW6QNLYN27CGTMGG5K2N2BQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:47.060058932 (525h41m46.931141967s) [upload=461126]: Next returned nil segment Dec 27 13:10:47.060073020 (525h41m46.931156065s) [upload=461126]: Waiting for error group managing segments... Dec 27 13:10:47.409551807 (525h41m47.280634842s) [noise=true , address=64.49.13.170:4724 , piece_id=FRNFGNNL57GYHDY2EB2DXRQEHUZQTKYYIS3475N7R54H5HATCXVQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.599519939 (525h41m47.470602974s) [noise=true , address=145.79.227.16:4932 , piece_id=425NEIP6DT3OBKARRSWY76MYMSAZ6VYUWDEY2TKQDJM66F6DQ2YQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.605459099 (525h41m47.476542144s) [noise=true , address=145.79.227.43:4162 , piece_id=5D3REI4QA47RUUN3IMIOW6CXFZZAZRYLKVAZCL6YR4MPG7UV4KPQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.707001222 (525h41m47.578084257s) [noise=true , address=145.79.227.34:4911 , piece_id=M3ODVIPMMESPK5PJTT2N7TX2N6WIZAMKH4U7HMEOVHDPQMWIS5ZA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.820931805 (525h41m47.69201485s) [noise=true , address=77.90.31.18:4833 , piece_id=TPMTEEA74KLTJ5V6YIMXCNHVM7CMXNAPI5JDH53LM3MBII42BGEA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.824299398 (525h41m47.695382433s) [noise=true , address=45.91.116.7:4020 , piece_id=TZMEUIWTKW7STBEA6VFITVALLVXHIED6C7JVPUYL73CSC2AG5YSA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.825288943 (525h41m47.696371978s) [noise=true , address=77.90.31.17:4792 , piece_id=QOZBFTVMOQ3CAF7BLJVABEIVKIKGXXW6QNLYN27CGTMGG5K2N2BQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.852715927 (525h41m47.723798972s) [noise=true , address=77.90.31.19:4863 , piece_id=6ZLPLFQDENXYKH5BPW3IAMJY7QIDI2SUIRARCBYKNDMOE5BLGTMA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.008917121 (525h41m47.880000166s) [noise=true , address=77.90.31.21:4907 , piece_id=KV4WOT4TC5QJEUEXWGZ6TPHMYALHLB3ZEOW35RZVFTCMGJVWH6ZA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.038473493 (525h41m47.909556538s) [noise=true , address=77.90.31.12:4624 , piece_id=XUYFLD7JCKYE2LQ66PVOOSRJV4NUZI3VQN6CMG6AUQJETYYN5BCA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.505751901 (525h41m48.376834936s) [noise=true , address=145.79.227.44:4228 , piece_id=TVURXWI4IKDIXJ34HYZ5D4B4FGR6U4FUCZXACIV2GNGDPO4SAOLQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.521156504 (525h41m48.392239539s) [noise=true , address=145.79.227.30:4796 , piece_id=ISVTXMY2AMJI3RDRURII7CX4R7PVPCW7ES7TZFMEV4TXPUU4RYVA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.545623223 (525h41m48.416706268s) [noise=true , address=77.90.31.16:4745 , piece_id=7K2OUZKFBFNXQMDWDNJHYAA3LIHCIA2EL7FLKMVKOL744V4JN6GQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.552502362 (525h41m48.423585397s) [noise=true , address=145.79.227.40:4078 , piece_id=5JTMTHVDQIR5XO7EBLLBQVVTYB47SQSEIL6F6Y7AZWNH2PKPOYYQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.552521197 (525h41m48.423604232s) [seg_pos={0 0}, upload=461126]: Segment reached optimal threshold of 14 pieces. Dec 27 13:10:48.552693505 (525h41m48.423776551s) [noise=true , address=64.49.13.169:4707 , piece_id=CMVRTRWEMPOFKBA6DJ7TOVSCM56IOGVDNKP327EGO2CREAYWF6WA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:43628->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:43628->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43628->64.49.13.169:4707: use of closed network connection Dec 27 13:10:48.552701241 (525h41m48.423784276s) [noise=true , address=64.49.13.169:4707 , piece_id=CMVRTRWEMPOFKBA6DJ7TOVSCM56IOGVDNKP327EGO2CREAYWF6WA, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553068299 (525h41m48.424151344s) [noise=true , address=77.90.31.15:4718 , piece_id=IHG7SIDROBAPTLWNBEXSIZYG5L4LHI7N4CEPY3APQZMEQJFRACRQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: expected piece hash; context canceled; EOF Dec 27 13:10:48.553078509 (525h41m48.424161554s) [noise=true , address=77.90.31.15:4718 , piece_id=IHG7SIDROBAPTLWNBEXSIZYG5L4LHI7N4CEPY3APQZMEQJFRACRQ, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553264022 (525h41m48.424347068s) [noise=true , address=77.90.31.13:4651 , piece_id=YWKS2JTDS5QYYPTEEOF54N7LKZGCA6UXQT2DNUA5KRZUXSJ73CZQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 27 13:10:48.553272158 (525h41m48.424355193s) [noise=true , address=77.90.31.13:4651 , piece_id=YWKS2JTDS5QYYPTEEOF54N7LKZGCA6UXQT2DNUA5KRZUXSJ73CZQ, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553429377 (525h41m48.424512412s) [noise=true , address=77.90.31.11:4565 , piece_id=N56S54JPLASZYOGS6VX6YK2ED6GWJDIKCI6TZPGIEFSA4LXEAZJA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: expected piece hash; context canceled; EOF Dec 27 13:10:48.553437763 (525h41m48.424520798s) [noise=true , address=77.90.31.11:4565 , piece_id=N56S54JPLASZYOGS6VX6YK2ED6GWJDIKCI6TZPGIEFSA4LXEAZJA, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553456298 (525h41m48.424539343s) [seg_pos={0 0}, upload=461126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:10:48.553546450 (525h41m48.424629495s) [upload=461126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:10:48.553553333 (525h41m48.424636388s) [upload=461126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:10:48.860753135 (525h41m48.73183618s) [upload=461126]: Done uploading segments... Dec 27 13:10:48.860772321 (525h41m48.731855356s) [upload=461126]: Upload finished. err: <nil> Duration: 42.247780121s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..........X.X......X..............X..X.X...X.XX..XX.X.......] duration: 3.410793407s trace: https://jaeger.dev.storj.io/trace/610bf000d9f46fe4 FAIL (2025-12-27 13:13:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.410816851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X..X.X..XX.X......X..X..XXX...........XX..X...XX......X..] duration: 6.011913059s trace: https://jaeger.dev.storj.io/trace/6263dbd7e0cc9887 FAIL (2025-12-27 13:13:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: WAB3DSTFEMGM2DD6Z2LM3LUZU4WTHHADLX4HFOOHH3PHDK3DGUCQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: ZJ7Z6TM4S6HZNLSQMYGCJF4GBZ37X2DPZLSTVL6OTZZ6XXQWQZUA) context deadline exceeded ======LOGS====== ments... Dec 27 13:11:54.603411583 (525h42m54.474494618s) [upload=461143]: Got next segment. Inline: false Dec 27 13:11:54.603464263 (525h42m54.474547298s) [upload=461143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:11:54.603469934 (525h42m54.474552969s) [upload=461143]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:11:54.851962066 (525h42m54.723045101s) [seg_pos={0 0}, upload=461143]: Begin upload segment... Dec 27 13:11:54.852007863 (525h42m54.723090898s) [seg_pos={0 0}, upload=461143]: Done begin upload segment. Dec 27 13:11:54.852036988 (525h42m54.723120033s) [noise=true , address=145.79.227.11:4732 , piece_id=32IZCSOVKNK4ZQ5TB667CWKXCDEUF374AAT2MUYELO2SI3EHVPZA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.852090079 (525h42m54.723173124s) [noise=true , address=45.91.116.11:4120 , piece_id=QT43X5RESASKXMN2MNG2AUUF2EFYTQNLGFYW63WKOVLXF363CL7A, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.852660696 (525h42m54.723743731s) [noise=true , address=145.79.227.7:4295 , piece_id=XSLKEJYGEVRW4DBJES6D4PV6RCMFSF7IAQMDFXIVBHGG5ROZPRAA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.853181688 (525h42m54.724264733s) [noise=true , address=145.79.227.37:4565 , piece_id=GQRCPNXYAGYI6OVIXEYJW6LBHHRW24RWEXQVOH7GLKNLKS2YEICA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.853298360 (525h42m54.724381395s) [noise=true , address=45.91.116.13:4156 , piece_id=ZJ7Z6TM4S6HZNLSQMYGCJF4GBZ37X2DPZLSTVL6OTZZ6XXQWQZUA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.853654037 (525h42m54.724737082s) [noise=true , address=145.79.227.4:4186 , piece_id=V7K5WZOTTJFYIDUOEALKUXICKPZNOE6BKUHBFG42LJZG3ZKZZ3AQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.853866481 (525h42m54.724949516s) [noise=true , address=145.79.227.8:4330 , piece_id=VSDGHHACDOJR6IRJQ4JFCMLGUMXJ3WFVCLDEGOFI4WWHEFXA6J7Q, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.854214113 (525h42m54.725297158s) [noise=true , address=145.79.227.12:4765 , piece_id=GI2JKEFEGSZ3MXHT46TUF5KCJXWMDE7KWNANCFFEKWEHLIL7JPEA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.854917122 (525h42m54.726000157s) [noise=true , address=77.90.31.10:4520 , piece_id=WAB3DSTFEMGM2DD6Z2LM3LUZU4WTHHADLX4HFOOHH3PHDK3DGUCQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.855180353 (525h42m54.726263419s) [noise=true , address=77.90.31.18:4808 , piece_id=LWFQVFUNZBH2OXHH5D6TJG6JYCTRXPVKZT2HDHZGFHR7Y3EW6INA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.855366588 (525h42m54.726449623s) [noise=true , address=45.91.116.12:4227 , piece_id=W32EAOOSG3GNI7RLN5AOSK6E3ELF2KIHQD2PRAEFAZ5KGG3DJ4FQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.855752082 (525h42m54.726835117s) [noise=true , address=145.79.227.46:4603 , piece_id=VFRQEFEREMCYBRM5JIITQBYLXGMMNU3Z4T2B5OOFPG7ZIEZYF7WA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.855786237 (525h42m54.726869282s) [noise=true , address=145.79.227.36:4517 , piece_id=2VDHJ2R3ZQN36XAO7VZDCGMZAEUZLV6VUS42EGYKRZ2QD7RCAVTQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.856306267 (525h42m54.727389312s) [noise=true , address=145.79.227.50:4374 , piece_id=DO5MBDGRJ4KB3754BITR332JW2PQRRVRYOOCZAWGRZL7UDX42SKQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.856787093 (525h42m54.727870138s) [noise=true , address=145.79.227.2:4114 , piece_id=E7V5VBZNUXXO256EY7B33FUK3WZFIWKK2SV4X5OQGYK2INDVLHWA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.857231369 (525h42m54.728314404s) [noise=true , address=145.79.227.39:4013 , piece_id=MEP3F5X65YYMTWJUOFQQ7UXEWHUJSGAW4LQL73PHOICN7JHTQPWQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:56.562238416 (525h42m56.433321462s) [upload=461143]: Next returned nil segment Dec 27 13:11:56.562272652 (525h42m56.433355707s) [upload=461143]: Waiting for error group managing segments... Dec 27 13:11:56.847194261 (525h42m56.718277296s) [noise=true , address=145.79.227.50:4374 , piece_id=DO5MBDGRJ4KB3754BITR332JW2PQRRVRYOOCZAWGRZL7UDX42SKQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.855082993 (525h42m56.726166038s) [noise=true , address=145.79.227.12:4765 , piece_id=GI2JKEFEGSZ3MXHT46TUF5KCJXWMDE7KWNANCFFEKWEHLIL7JPEA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.859782672 (525h42m56.730865707s) [noise=true , address=145.79.227.7:4295 , piece_id=XSLKEJYGEVRW4DBJES6D4PV6RCMFSF7IAQMDFXIVBHGG5ROZPRAA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.862445593 (525h42m56.733528638s) [noise=true , address=145.79.227.11:4732 , piece_id=32IZCSOVKNK4ZQ5TB667CWKXCDEUF374AAT2MUYELO2SI3EHVPZA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.864105313 (525h42m56.735188348s) [noise=true , address=145.79.227.2:4114 , piece_id=E7V5VBZNUXXO256EY7B33FUK3WZFIWKK2SV4X5OQGYK2INDVLHWA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.865877738 (525h42m56.736960783s) [noise=true , address=145.79.227.39:4013 , piece_id=MEP3F5X65YYMTWJUOFQQ7UXEWHUJSGAW4LQL73PHOICN7JHTQPWQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.874552456 (525h42m56.745635501s) [noise=true , address=145.79.227.8:4330 , piece_id=VSDGHHACDOJR6IRJQ4JFCMLGUMXJ3WFVCLDEGOFI4WWHEFXA6J7Q, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.884004645 (525h42m56.75508769s) [noise=true , address=145.79.227.4:4186 , piece_id=V7K5WZOTTJFYIDUOEALKUXICKPZNOE6BKUHBFG42LJZG3ZKZZ3AQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.888523760 (525h42m56.759606795s) [noise=true , address=145.79.227.36:4517 , piece_id=2VDHJ2R3ZQN36XAO7VZDCGMZAEUZLV6VUS42EGYKRZ2QD7RCAVTQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.888808272 (525h42m56.759891307s) [noise=true , address=145.79.227.37:4565 , piece_id=GQRCPNXYAGYI6OVIXEYJW6LBHHRW24RWEXQVOH7GLKNLKS2YEICA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.891212600 (525h42m56.762295635s) [noise=true , address=145.79.227.46:4603 , piece_id=VFRQEFEREMCYBRM5JIITQBYLXGMMNU3Z4T2B5OOFPG7ZIEZYF7WA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:57.154988905 (525h42m57.02607195s) [noise=true , address=45.91.116.13:4156 , piece_id=ZJ7Z6TM4S6HZNLSQMYGCJF4GBZ37X2DPZLSTVL6OTZZ6XXQWQZUA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:57.261361597 (525h42m57.132444632s) [noise=true , address=77.90.31.10:4520 , piece_id=WAB3DSTFEMGM2DD6Z2LM3LUZU4WTHHADLX4HFOOHH3PHDK3DGUCQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:57.261385273 (525h42m57.132468308s) [seg_pos={0 0}, upload=461143]: Segment reached optimal threshold of 13 pieces. Dec 27 13:11:57.261576646 (525h42m57.132659682s) [noise=true , address=77.90.31.18:4808 , piece_id=LWFQVFUNZBH2OXHH5D6TJG6JYCTRXPVKZT2HDHZGFHR7Y3EW6INA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:37990->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:37990->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37990->77.90.31.18:4808: use of closed network connection Dec 27 13:11:57.261588308 (525h42m57.132671353s) [noise=true , address=77.90.31.18:4808 , piece_id=LWFQVFUNZBH2OXHH5D6TJG6JYCTRXPVKZT2HDHZGFHR7Y3EW6INA, seg_pos={0 0}, upload=461143]: Long tail cancellation due to optimal threshold Dec 27 13:11:57.261765857 (525h42m57.132848892s) [noise=true , address=45.91.116.12:4227 , piece_id=W32EAOOSG3GNI7RLN5AOSK6E3ELF2KIHQD2PRAEFAZ5KGG3DJ4FQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:37366->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:37366->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37366->45.91.116.12:4227: use of closed network connection Dec 27 13:11:57.261787288 (525h42m57.132870333s) [noise=true , address=45.91.116.12:4227 , piece_id=W32EAOOSG3GNI7RLN5AOSK6E3ELF2KIHQD2PRAEFAZ5KGG3DJ4FQ, seg_pos={0 0}, upload=461143]: Long tail cancellation due to optimal threshold Dec 27 13:11:57.261864264 (525h42m57.132947379s) [noise=true , address=45.91.116.11:4120 , piece_id=QT43X5RESASKXMN2MNG2AUUF2EFYTQNLGFYW63WKOVLXF363CL7A, seg_pos={0 0}, upload=461143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 13:11:57.261903109 (525h42m57.132986154s) [noise=true , address=45.91.116.11:4120 , piece_id=QT43X5RESASKXMN2MNG2AUUF2EFYTQNLGFYW63WKOVLXF363CL7A, seg_pos={0 0}, upload=461143]: Long tail cancellation due to optimal threshold Dec 27 13:11:57.261984373 (525h42m57.133067408s) [seg_pos={0 0}, upload=461143]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:11:57.262208980 (525h42m57.133292015s) [upload=461143]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:11:57.262218149 (525h42m57.133301194s) [upload=461143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:11:57.551171545 (525h42m57.42225458s) [upload=461143]: Done uploading segments... Dec 27 13:11:57.551195481 (525h42m57.422278516s) [upload=461143]: Upload finished. err: <nil> Duration: 6.011944808s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...X..X.X..XX.X......X..X..XXX...........XX..X...XX......X.] duration: 5.685506654s trace: https://jaeger.dev.storj.io/trace/6263dbd7e0cc9887 FAIL (2025-12-27 13:12:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: WAB3DSTFEMGM2DD6Z2LM3LUZU4WTHHADLX4HFOOHH3PHDK3DGUCQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: ZJ7Z6TM4S6HZNLSQMYGCJF4GBZ37X2DPZLSTVL6OTZZ6XXQWQZUA) context deadline exceeded ======LOGS====== ments... Dec 27 13:11:54.603411583 (525h42m54.474494618s) [upload=461143]: Got next segment. Inline: false Dec 27 13:11:54.603464263 (525h42m54.474547298s) [upload=461143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:11:54.603469934 (525h42m54.474552969s) [upload=461143]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:11:54.851962066 (525h42m54.723045101s) [seg_pos={0 0}, upload=461143]: Begin upload segment... Dec 27 13:11:54.852007863 (525h42m54.723090898s) [seg_pos={0 0}, upload=461143]: Done begin upload segment. Dec 27 13:11:54.852036988 (525h42m54.723120033s) [noise=true , address=145.79.227.11:4732 , piece_id=32IZCSOVKNK4ZQ5TB667CWKXCDEUF374AAT2MUYELO2SI3EHVPZA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.852090079 (525h42m54.723173124s) [noise=true , address=45.91.116.11:4120 , piece_id=QT43X5RESASKXMN2MNG2AUUF2EFYTQNLGFYW63WKOVLXF363CL7A, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.852660696 (525h42m54.723743731s) [noise=true , address=145.79.227.7:4295 , piece_id=XSLKEJYGEVRW4DBJES6D4PV6RCMFSF7IAQMDFXIVBHGG5ROZPRAA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.853181688 (525h42m54.724264733s) [noise=true , address=145.79.227.37:4565 , piece_id=GQRCPNXYAGYI6OVIXEYJW6LBHHRW24RWEXQVOH7GLKNLKS2YEICA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.853298360 (525h42m54.724381395s) [noise=true , address=45.91.116.13:4156 , piece_id=ZJ7Z6TM4S6HZNLSQMYGCJF4GBZ37X2DPZLSTVL6OTZZ6XXQWQZUA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.853654037 (525h42m54.724737082s) [noise=true , address=145.79.227.4:4186 , piece_id=V7K5WZOTTJFYIDUOEALKUXICKPZNOE6BKUHBFG42LJZG3ZKZZ3AQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.853866481 (525h42m54.724949516s) [noise=true , address=145.79.227.8:4330 , piece_id=VSDGHHACDOJR6IRJQ4JFCMLGUMXJ3WFVCLDEGOFI4WWHEFXA6J7Q, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.854214113 (525h42m54.725297158s) [noise=true , address=145.79.227.12:4765 , piece_id=GI2JKEFEGSZ3MXHT46TUF5KCJXWMDE7KWNANCFFEKWEHLIL7JPEA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.854917122 (525h42m54.726000157s) [noise=true , address=77.90.31.10:4520 , piece_id=WAB3DSTFEMGM2DD6Z2LM3LUZU4WTHHADLX4HFOOHH3PHDK3DGUCQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.855180353 (525h42m54.726263419s) [noise=true , address=77.90.31.18:4808 , piece_id=LWFQVFUNZBH2OXHH5D6TJG6JYCTRXPVKZT2HDHZGFHR7Y3EW6INA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.855366588 (525h42m54.726449623s) [noise=true , address=45.91.116.12:4227 , piece_id=W32EAOOSG3GNI7RLN5AOSK6E3ELF2KIHQD2PRAEFAZ5KGG3DJ4FQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.855752082 (525h42m54.726835117s) [noise=true , address=145.79.227.46:4603 , piece_id=VFRQEFEREMCYBRM5JIITQBYLXGMMNU3Z4T2B5OOFPG7ZIEZYF7WA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.855786237 (525h42m54.726869282s) [noise=true , address=145.79.227.36:4517 , piece_id=2VDHJ2R3ZQN36XAO7VZDCGMZAEUZLV6VUS42EGYKRZ2QD7RCAVTQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.856306267 (525h42m54.727389312s) [noise=true , address=145.79.227.50:4374 , piece_id=DO5MBDGRJ4KB3754BITR332JW2PQRRVRYOOCZAWGRZL7UDX42SKQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.856787093 (525h42m54.727870138s) [noise=true , address=145.79.227.2:4114 , piece_id=E7V5VBZNUXXO256EY7B33FUK3WZFIWKK2SV4X5OQGYK2INDVLHWA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.857231369 (525h42m54.728314404s) [noise=true , address=145.79.227.39:4013 , piece_id=MEP3F5X65YYMTWJUOFQQ7UXEWHUJSGAW4LQL73PHOICN7JHTQPWQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:56.562238416 (525h42m56.433321462s) [upload=461143]: Next returned nil segment Dec 27 13:11:56.562272652 (525h42m56.433355707s) [upload=461143]: Waiting for error group managing segments... Dec 27 13:11:56.847194261 (525h42m56.718277296s) [noise=true , address=145.79.227.50:4374 , piece_id=DO5MBDGRJ4KB3754BITR332JW2PQRRVRYOOCZAWGRZL7UDX42SKQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.855082993 (525h42m56.726166038s) [noise=true , address=145.79.227.12:4765 , piece_id=GI2JKEFEGSZ3MXHT46TUF5KCJXWMDE7KWNANCFFEKWEHLIL7JPEA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.859782672 (525h42m56.730865707s) [noise=true , address=145.79.227.7:4295 , piece_id=XSLKEJYGEVRW4DBJES6D4PV6RCMFSF7IAQMDFXIVBHGG5ROZPRAA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.862445593 (525h42m56.733528638s) [noise=true , address=145.79.227.11:4732 , piece_id=32IZCSOVKNK4ZQ5TB667CWKXCDEUF374AAT2MUYELO2SI3EHVPZA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.864105313 (525h42m56.735188348s) [noise=true , address=145.79.227.2:4114 , piece_id=E7V5VBZNUXXO256EY7B33FUK3WZFIWKK2SV4X5OQGYK2INDVLHWA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.865877738 (525h42m56.736960783s) [noise=true , address=145.79.227.39:4013 , piece_id=MEP3F5X65YYMTWJUOFQQ7UXEWHUJSGAW4LQL73PHOICN7JHTQPWQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.874552456 (525h42m56.745635501s) [noise=true , address=145.79.227.8:4330 , piece_id=VSDGHHACDOJR6IRJQ4JFCMLGUMXJ3WFVCLDEGOFI4WWHEFXA6J7Q, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.884004645 (525h42m56.75508769s) [noise=true , address=145.79.227.4:4186 , piece_id=V7K5WZOTTJFYIDUOEALKUXICKPZNOE6BKUHBFG42LJZG3ZKZZ3AQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.888523760 (525h42m56.759606795s) [noise=true , address=145.79.227.36:4517 , piece_id=2VDHJ2R3ZQN36XAO7VZDCGMZAEUZLV6VUS42EGYKRZ2QD7RCAVTQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.888808272 (525h42m56.759891307s) [noise=true , address=145.79.227.37:4565 , piece_id=GQRCPNXYAGYI6OVIXEYJW6LBHHRW24RWEXQVOH7GLKNLKS2YEICA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.891212600 (525h42m56.762295635s) [noise=true , address=145.79.227.46:4603 , piece_id=VFRQEFEREMCYBRM5JIITQBYLXGMMNU3Z4T2B5OOFPG7ZIEZYF7WA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:57.154988905 (525h42m57.02607195s) [noise=true , address=45.91.116.13:4156 , piece_id=ZJ7Z6TM4S6HZNLSQMYGCJF4GBZ37X2DPZLSTVL6OTZZ6XXQWQZUA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:57.261361597 (525h42m57.132444632s) [noise=true , address=77.90.31.10:4520 , piece_id=WAB3DSTFEMGM2DD6Z2LM3LUZU4WTHHADLX4HFOOHH3PHDK3DGUCQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:57.261385273 (525h42m57.132468308s) [seg_pos={0 0}, upload=461143]: Segment reached optimal threshold of 13 pieces. Dec 27 13:11:57.261576646 (525h42m57.132659682s) [noise=true , address=77.90.31.18:4808 , piece_id=LWFQVFUNZBH2OXHH5D6TJG6JYCTRXPVKZT2HDHZGFHR7Y3EW6INA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:37990->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:37990->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37990->77.90.31.18:4808: use of closed network connection Dec 27 13:11:57.261588308 (525h42m57.132671353s) [noise=true , address=77.90.31.18:4808 , piece_id=LWFQVFUNZBH2OXHH5D6TJG6JYCTRXPVKZT2HDHZGFHR7Y3EW6INA, seg_pos={0 0}, upload=461143]: Long tail cancellation due to optimal threshold Dec 27 13:11:57.261765857 (525h42m57.132848892s) [noise=true , address=45.91.116.12:4227 , piece_id=W32EAOOSG3GNI7RLN5AOSK6E3ELF2KIHQD2PRAEFAZ5KGG3DJ4FQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:37366->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:37366->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37366->45.91.116.12:4227: use of closed network connection Dec 27 13:11:57.261787288 (525h42m57.132870333s) [noise=true , address=45.91.116.12:4227 , piece_id=W32EAOOSG3GNI7RLN5AOSK6E3ELF2KIHQD2PRAEFAZ5KGG3DJ4FQ, seg_pos={0 0}, upload=461143]: Long tail cancellation due to optimal threshold Dec 27 13:11:57.261864264 (525h42m57.132947379s) [noise=true , address=45.91.116.11:4120 , piece_id=QT43X5RESASKXMN2MNG2AUUF2EFYTQNLGFYW63WKOVLXF363CL7A, seg_pos={0 0}, upload=461143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 13:11:57.261903109 (525h42m57.132986154s) [noise=true , address=45.91.116.11:4120 , piece_id=QT43X5RESASKXMN2MNG2AUUF2EFYTQNLGFYW63WKOVLXF363CL7A, seg_pos={0 0}, upload=461143]: Long tail cancellation due to optimal threshold Dec 27 13:11:57.261984373 (525h42m57.133067408s) [seg_pos={0 0}, upload=461143]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:11:57.262208980 (525h42m57.133292015s) [upload=461143]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:11:57.262218149 (525h42m57.133301194s) [upload=461143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:11:57.551171545 (525h42m57.42225458s) [upload=461143]: Done uploading segments... Dec 27 13:11:57.551195481 (525h42m57.422278516s) [upload=461143]: Upload finished. err: <nil> Duration: 5.685524658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.............................X.] duration: 43.953611296s trace: https://jaeger.dev.storj.io/trace/15e7e71b1e4a765b FAIL (2025-12-27 13:12:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: TPMTEEA74KLTJ5V6YIMXCNHVM7CMXNAPI5JDH53LM3MBII42BGEA) context deadline exceeded; piecestore: (Node ID: 1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, Piece ID: TZMEUIWTKW7STBEA6VFITVALLVXHIED6C7JVPUYL73CSC2AG5YSA) context deadline exceeded; piecestore: (Node ID: 12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, Piece ID: QOZBFTVMOQ3CAF7BLJVABEIVKIKGXXW6QNLYN27CGTMGG5K2N2BQ) context deadline exceeded ======LOGS====== oise=true , address=77.90.31.13:4651 , piece_id=YWKS2JTDS5QYYPTEEOF54N7LKZGCA6UXQT2DNUA5KRZUXSJ73CZQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.709860762 (525h41m45.580943797s) [noise=true , address=64.49.13.170:4724 , piece_id=FRNFGNNL57GYHDY2EB2DXRQEHUZQTKYYIS3475N7R54H5HATCXVQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.710306931 (525h41m45.581389976s) [noise=true , address=145.79.227.34:4911 , piece_id=M3ODVIPMMESPK5PJTT2N7TX2N6WIZAMKH4U7HMEOVHDPQMWIS5ZA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.710677497 (525h41m45.581760532s) [noise=true , address=77.90.31.12:4624 , piece_id=XUYFLD7JCKYE2LQ66PVOOSRJV4NUZI3VQN6CMG6AUQJETYYN5BCA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.711052121 (525h41m45.582135156s) [noise=true , address=77.90.31.19:4863 , piece_id=6ZLPLFQDENXYKH5BPW3IAMJY7QIDI2SUIRARCBYKNDMOE5BLGTMA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.711430001 (525h41m45.582513036s) [noise=true , address=145.79.227.16:4932 , piece_id=425NEIP6DT3OBKARRSWY76MYMSAZ6VYUWDEY2TKQDJM66F6DQ2YQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.711825103 (525h41m45.582908138s) [noise=true , address=77.90.31.18:4833 , piece_id=TPMTEEA74KLTJ5V6YIMXCNHVM7CMXNAPI5JDH53LM3MBII42BGEA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.712248579 (525h41m45.583331624s) [noise=true , address=145.79.227.40:4078 , piece_id=5JTMTHVDQIR5XO7EBLLBQVVTYB47SQSEIL6F6Y7AZWNH2PKPOYYQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.712670562 (525h41m45.583753607s) [noise=true , address=77.90.31.11:4565 , piece_id=N56S54JPLASZYOGS6VX6YK2ED6GWJDIKCI6TZPGIEFSA4LXEAZJA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.713324337 (525h41m45.584407372s) [noise=true , address=64.49.13.169:4707 , piece_id=CMVRTRWEMPOFKBA6DJ7TOVSCM56IOGVDNKP327EGO2CREAYWF6WA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.713941212 (525h41m45.585024247s) [noise=true , address=145.79.227.43:4162 , piece_id=5D3REI4QA47RUUN3IMIOW6CXFZZAZRYLKVAZCL6YR4MPG7UV4KPQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.715054432 (525h41m45.586137467s) [noise=true , address=77.90.31.15:4718 , piece_id=IHG7SIDROBAPTLWNBEXSIZYG5L4LHI7N4CEPY3APQZMEQJFRACRQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.715571597 (525h41m45.586654632s) [noise=true , address=45.91.116.7:4020 , piece_id=TZMEUIWTKW7STBEA6VFITVALLVXHIED6C7JVPUYL73CSC2AG5YSA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.716503260 (525h41m45.587586305s) [noise=true , address=77.90.31.21:4907 , piece_id=KV4WOT4TC5QJEUEXWGZ6TPHMYALHLB3ZEOW35RZVFTCMGJVWH6ZA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.717147277 (525h41m45.588230312s) [noise=true , address=145.79.227.30:4796 , piece_id=ISVTXMY2AMJI3RDRURII7CX4R7PVPCW7ES7TZFMEV4TXPUU4RYVA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.718560367 (525h41m45.589643402s) [noise=true , address=77.90.31.17:4792 , piece_id=QOZBFTVMOQ3CAF7BLJVABEIVKIKGXXW6QNLYN27CGTMGG5K2N2BQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:47.060058932 (525h41m46.931141967s) [upload=461126]: Next returned nil segment Dec 27 13:10:47.060073020 (525h41m46.931156065s) [upload=461126]: Waiting for error group managing segments... Dec 27 13:10:47.409551807 (525h41m47.280634842s) [noise=true , address=64.49.13.170:4724 , piece_id=FRNFGNNL57GYHDY2EB2DXRQEHUZQTKYYIS3475N7R54H5HATCXVQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.599519939 (525h41m47.470602974s) [noise=true , address=145.79.227.16:4932 , piece_id=425NEIP6DT3OBKARRSWY76MYMSAZ6VYUWDEY2TKQDJM66F6DQ2YQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.605459099 (525h41m47.476542144s) [noise=true , address=145.79.227.43:4162 , piece_id=5D3REI4QA47RUUN3IMIOW6CXFZZAZRYLKVAZCL6YR4MPG7UV4KPQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.707001222 (525h41m47.578084257s) [noise=true , address=145.79.227.34:4911 , piece_id=M3ODVIPMMESPK5PJTT2N7TX2N6WIZAMKH4U7HMEOVHDPQMWIS5ZA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.820931805 (525h41m47.69201485s) [noise=true , address=77.90.31.18:4833 , piece_id=TPMTEEA74KLTJ5V6YIMXCNHVM7CMXNAPI5JDH53LM3MBII42BGEA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.824299398 (525h41m47.695382433s) [noise=true , address=45.91.116.7:4020 , piece_id=TZMEUIWTKW7STBEA6VFITVALLVXHIED6C7JVPUYL73CSC2AG5YSA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.825288943 (525h41m47.696371978s) [noise=true , address=77.90.31.17:4792 , piece_id=QOZBFTVMOQ3CAF7BLJVABEIVKIKGXXW6QNLYN27CGTMGG5K2N2BQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.852715927 (525h41m47.723798972s) [noise=true , address=77.90.31.19:4863 , piece_id=6ZLPLFQDENXYKH5BPW3IAMJY7QIDI2SUIRARCBYKNDMOE5BLGTMA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.008917121 (525h41m47.880000166s) [noise=true , address=77.90.31.21:4907 , piece_id=KV4WOT4TC5QJEUEXWGZ6TPHMYALHLB3ZEOW35RZVFTCMGJVWH6ZA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.038473493 (525h41m47.909556538s) [noise=true , address=77.90.31.12:4624 , piece_id=XUYFLD7JCKYE2LQ66PVOOSRJV4NUZI3VQN6CMG6AUQJETYYN5BCA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.505751901 (525h41m48.376834936s) [noise=true , address=145.79.227.44:4228 , piece_id=TVURXWI4IKDIXJ34HYZ5D4B4FGR6U4FUCZXACIV2GNGDPO4SAOLQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.521156504 (525h41m48.392239539s) [noise=true , address=145.79.227.30:4796 , piece_id=ISVTXMY2AMJI3RDRURII7CX4R7PVPCW7ES7TZFMEV4TXPUU4RYVA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.545623223 (525h41m48.416706268s) [noise=true , address=77.90.31.16:4745 , piece_id=7K2OUZKFBFNXQMDWDNJHYAA3LIHCIA2EL7FLKMVKOL744V4JN6GQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.552502362 (525h41m48.423585397s) [noise=true , address=145.79.227.40:4078 , piece_id=5JTMTHVDQIR5XO7EBLLBQVVTYB47SQSEIL6F6Y7AZWNH2PKPOYYQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.552521197 (525h41m48.423604232s) [seg_pos={0 0}, upload=461126]: Segment reached optimal threshold of 14 pieces. Dec 27 13:10:48.552693505 (525h41m48.423776551s) [noise=true , address=64.49.13.169:4707 , piece_id=CMVRTRWEMPOFKBA6DJ7TOVSCM56IOGVDNKP327EGO2CREAYWF6WA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:43628->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:43628->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43628->64.49.13.169:4707: use of closed network connection Dec 27 13:10:48.552701241 (525h41m48.423784276s) [noise=true , address=64.49.13.169:4707 , piece_id=CMVRTRWEMPOFKBA6DJ7TOVSCM56IOGVDNKP327EGO2CREAYWF6WA, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553068299 (525h41m48.424151344s) [noise=true , address=77.90.31.15:4718 , piece_id=IHG7SIDROBAPTLWNBEXSIZYG5L4LHI7N4CEPY3APQZMEQJFRACRQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: expected piece hash; context canceled; EOF Dec 27 13:10:48.553078509 (525h41m48.424161554s) [noise=true , address=77.90.31.15:4718 , piece_id=IHG7SIDROBAPTLWNBEXSIZYG5L4LHI7N4CEPY3APQZMEQJFRACRQ, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553264022 (525h41m48.424347068s) [noise=true , address=77.90.31.13:4651 , piece_id=YWKS2JTDS5QYYPTEEOF54N7LKZGCA6UXQT2DNUA5KRZUXSJ73CZQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 27 13:10:48.553272158 (525h41m48.424355193s) [noise=true , address=77.90.31.13:4651 , piece_id=YWKS2JTDS5QYYPTEEOF54N7LKZGCA6UXQT2DNUA5KRZUXSJ73CZQ, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553429377 (525h41m48.424512412s) [noise=true , address=77.90.31.11:4565 , piece_id=N56S54JPLASZYOGS6VX6YK2ED6GWJDIKCI6TZPGIEFSA4LXEAZJA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: expected piece hash; context canceled; EOF Dec 27 13:10:48.553437763 (525h41m48.424520798s) [noise=true , address=77.90.31.11:4565 , piece_id=N56S54JPLASZYOGS6VX6YK2ED6GWJDIKCI6TZPGIEFSA4LXEAZJA, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553456298 (525h41m48.424539343s) [seg_pos={0 0}, upload=461126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:10:48.553546450 (525h41m48.424629495s) [upload=461126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:10:48.553553333 (525h41m48.424636388s) [upload=461126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:10:48.860753135 (525h41m48.73183618s) [upload=461126]: Done uploading segments... Dec 27 13:10:48.860772321 (525h41m48.731855356s) [upload=461126]: Upload finished. err: <nil> Duration: 43.953629109s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...........X.X......X..............X..X.X...X.XX..XX.X......] duration: 3.425437303s trace: https://jaeger.dev.storj.io/trace/610bf000d9f46fe4 FAIL (2025-12-27 13:12:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.425459716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX...X..X.X..XX.X......X..X..XXX...........XX..X...XX......X] duration: 30.003061395s trace: https://jaeger.dev.storj.io/trace/6263dbd7e0cc9887 FAIL (2025-12-27 13:12:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: WAB3DSTFEMGM2DD6Z2LM3LUZU4WTHHADLX4HFOOHH3PHDK3DGUCQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: ZJ7Z6TM4S6HZNLSQMYGCJF4GBZ37X2DPZLSTVL6OTZZ6XXQWQZUA) context deadline exceeded ======LOGS====== ments... Dec 27 13:11:54.603411583 (525h42m54.474494618s) [upload=461143]: Got next segment. Inline: false Dec 27 13:11:54.603464263 (525h42m54.474547298s) [upload=461143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:11:54.603469934 (525h42m54.474552969s) [upload=461143]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:11:54.851962066 (525h42m54.723045101s) [seg_pos={0 0}, upload=461143]: Begin upload segment... Dec 27 13:11:54.852007863 (525h42m54.723090898s) [seg_pos={0 0}, upload=461143]: Done begin upload segment. Dec 27 13:11:54.852036988 (525h42m54.723120033s) [noise=true , address=145.79.227.11:4732 , piece_id=32IZCSOVKNK4ZQ5TB667CWKXCDEUF374AAT2MUYELO2SI3EHVPZA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.852090079 (525h42m54.723173124s) [noise=true , address=45.91.116.11:4120 , piece_id=QT43X5RESASKXMN2MNG2AUUF2EFYTQNLGFYW63WKOVLXF363CL7A, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.852660696 (525h42m54.723743731s) [noise=true , address=145.79.227.7:4295 , piece_id=XSLKEJYGEVRW4DBJES6D4PV6RCMFSF7IAQMDFXIVBHGG5ROZPRAA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.853181688 (525h42m54.724264733s) [noise=true , address=145.79.227.37:4565 , piece_id=GQRCPNXYAGYI6OVIXEYJW6LBHHRW24RWEXQVOH7GLKNLKS2YEICA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.853298360 (525h42m54.724381395s) [noise=true , address=45.91.116.13:4156 , piece_id=ZJ7Z6TM4S6HZNLSQMYGCJF4GBZ37X2DPZLSTVL6OTZZ6XXQWQZUA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.853654037 (525h42m54.724737082s) [noise=true , address=145.79.227.4:4186 , piece_id=V7K5WZOTTJFYIDUOEALKUXICKPZNOE6BKUHBFG42LJZG3ZKZZ3AQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.853866481 (525h42m54.724949516s) [noise=true , address=145.79.227.8:4330 , piece_id=VSDGHHACDOJR6IRJQ4JFCMLGUMXJ3WFVCLDEGOFI4WWHEFXA6J7Q, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.854214113 (525h42m54.725297158s) [noise=true , address=145.79.227.12:4765 , piece_id=GI2JKEFEGSZ3MXHT46TUF5KCJXWMDE7KWNANCFFEKWEHLIL7JPEA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.854917122 (525h42m54.726000157s) [noise=true , address=77.90.31.10:4520 , piece_id=WAB3DSTFEMGM2DD6Z2LM3LUZU4WTHHADLX4HFOOHH3PHDK3DGUCQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.855180353 (525h42m54.726263419s) [noise=true , address=77.90.31.18:4808 , piece_id=LWFQVFUNZBH2OXHH5D6TJG6JYCTRXPVKZT2HDHZGFHR7Y3EW6INA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.855366588 (525h42m54.726449623s) [noise=true , address=45.91.116.12:4227 , piece_id=W32EAOOSG3GNI7RLN5AOSK6E3ELF2KIHQD2PRAEFAZ5KGG3DJ4FQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.855752082 (525h42m54.726835117s) [noise=true , address=145.79.227.46:4603 , piece_id=VFRQEFEREMCYBRM5JIITQBYLXGMMNU3Z4T2B5OOFPG7ZIEZYF7WA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.855786237 (525h42m54.726869282s) [noise=true , address=145.79.227.36:4517 , piece_id=2VDHJ2R3ZQN36XAO7VZDCGMZAEUZLV6VUS42EGYKRZ2QD7RCAVTQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.856306267 (525h42m54.727389312s) [noise=true , address=145.79.227.50:4374 , piece_id=DO5MBDGRJ4KB3754BITR332JW2PQRRVRYOOCZAWGRZL7UDX42SKQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.856787093 (525h42m54.727870138s) [noise=true , address=145.79.227.2:4114 , piece_id=E7V5VBZNUXXO256EY7B33FUK3WZFIWKK2SV4X5OQGYK2INDVLHWA, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:54.857231369 (525h42m54.728314404s) [noise=true , address=145.79.227.39:4013 , piece_id=MEP3F5X65YYMTWJUOFQQ7UXEWHUJSGAW4LQL73PHOICN7JHTQPWQ, seg_pos={0 0}, upload=461143]: Uploading piece... Dec 27 13:11:56.562238416 (525h42m56.433321462s) [upload=461143]: Next returned nil segment Dec 27 13:11:56.562272652 (525h42m56.433355707s) [upload=461143]: Waiting for error group managing segments... Dec 27 13:11:56.847194261 (525h42m56.718277296s) [noise=true , address=145.79.227.50:4374 , piece_id=DO5MBDGRJ4KB3754BITR332JW2PQRRVRYOOCZAWGRZL7UDX42SKQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.855082993 (525h42m56.726166038s) [noise=true , address=145.79.227.12:4765 , piece_id=GI2JKEFEGSZ3MXHT46TUF5KCJXWMDE7KWNANCFFEKWEHLIL7JPEA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.859782672 (525h42m56.730865707s) [noise=true , address=145.79.227.7:4295 , piece_id=XSLKEJYGEVRW4DBJES6D4PV6RCMFSF7IAQMDFXIVBHGG5ROZPRAA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.862445593 (525h42m56.733528638s) [noise=true , address=145.79.227.11:4732 , piece_id=32IZCSOVKNK4ZQ5TB667CWKXCDEUF374AAT2MUYELO2SI3EHVPZA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.864105313 (525h42m56.735188348s) [noise=true , address=145.79.227.2:4114 , piece_id=E7V5VBZNUXXO256EY7B33FUK3WZFIWKK2SV4X5OQGYK2INDVLHWA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.865877738 (525h42m56.736960783s) [noise=true , address=145.79.227.39:4013 , piece_id=MEP3F5X65YYMTWJUOFQQ7UXEWHUJSGAW4LQL73PHOICN7JHTQPWQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.874552456 (525h42m56.745635501s) [noise=true , address=145.79.227.8:4330 , piece_id=VSDGHHACDOJR6IRJQ4JFCMLGUMXJ3WFVCLDEGOFI4WWHEFXA6J7Q, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.884004645 (525h42m56.75508769s) [noise=true , address=145.79.227.4:4186 , piece_id=V7K5WZOTTJFYIDUOEALKUXICKPZNOE6BKUHBFG42LJZG3ZKZZ3AQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.888523760 (525h42m56.759606795s) [noise=true , address=145.79.227.36:4517 , piece_id=2VDHJ2R3ZQN36XAO7VZDCGMZAEUZLV6VUS42EGYKRZ2QD7RCAVTQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.888808272 (525h42m56.759891307s) [noise=true , address=145.79.227.37:4565 , piece_id=GQRCPNXYAGYI6OVIXEYJW6LBHHRW24RWEXQVOH7GLKNLKS2YEICA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:56.891212600 (525h42m56.762295635s) [noise=true , address=145.79.227.46:4603 , piece_id=VFRQEFEREMCYBRM5JIITQBYLXGMMNU3Z4T2B5OOFPG7ZIEZYF7WA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:57.154988905 (525h42m57.02607195s) [noise=true , address=45.91.116.13:4156 , piece_id=ZJ7Z6TM4S6HZNLSQMYGCJF4GBZ37X2DPZLSTVL6OTZZ6XXQWQZUA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:57.261361597 (525h42m57.132444632s) [noise=true , address=77.90.31.10:4520 , piece_id=WAB3DSTFEMGM2DD6Z2LM3LUZU4WTHHADLX4HFOOHH3PHDK3DGUCQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: <nil> Dec 27 13:11:57.261385273 (525h42m57.132468308s) [seg_pos={0 0}, upload=461143]: Segment reached optimal threshold of 13 pieces. Dec 27 13:11:57.261576646 (525h42m57.132659682s) [noise=true , address=77.90.31.18:4808 , piece_id=LWFQVFUNZBH2OXHH5D6TJG6JYCTRXPVKZT2HDHZGFHR7Y3EW6INA, seg_pos={0 0}, upload=461143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:37990->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:37990->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37990->77.90.31.18:4808: use of closed network connection Dec 27 13:11:57.261588308 (525h42m57.132671353s) [noise=true , address=77.90.31.18:4808 , piece_id=LWFQVFUNZBH2OXHH5D6TJG6JYCTRXPVKZT2HDHZGFHR7Y3EW6INA, seg_pos={0 0}, upload=461143]: Long tail cancellation due to optimal threshold Dec 27 13:11:57.261765857 (525h42m57.132848892s) [noise=true , address=45.91.116.12:4227 , piece_id=W32EAOOSG3GNI7RLN5AOSK6E3ELF2KIHQD2PRAEFAZ5KGG3DJ4FQ, seg_pos={0 0}, upload=461143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:37366->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:37366->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37366->45.91.116.12:4227: use of closed network connection Dec 27 13:11:57.261787288 (525h42m57.132870333s) [noise=true , address=45.91.116.12:4227 , piece_id=W32EAOOSG3GNI7RLN5AOSK6E3ELF2KIHQD2PRAEFAZ5KGG3DJ4FQ, seg_pos={0 0}, upload=461143]: Long tail cancellation due to optimal threshold Dec 27 13:11:57.261864264 (525h42m57.132947379s) [noise=true , address=45.91.116.11:4120 , piece_id=QT43X5RESASKXMN2MNG2AUUF2EFYTQNLGFYW63WKOVLXF363CL7A, seg_pos={0 0}, upload=461143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 13:11:57.261903109 (525h42m57.132986154s) [noise=true , address=45.91.116.11:4120 , piece_id=QT43X5RESASKXMN2MNG2AUUF2EFYTQNLGFYW63WKOVLXF363CL7A, seg_pos={0 0}, upload=461143]: Long tail cancellation due to optimal threshold Dec 27 13:11:57.261984373 (525h42m57.133067408s) [seg_pos={0 0}, upload=461143]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:11:57.262208980 (525h42m57.133292015s) [upload=461143]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:11:57.262218149 (525h42m57.133301194s) [upload=461143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:11:57.551171545 (525h42m57.42225458s) [upload=461143]: Done uploading segments... Dec 27 13:11:57.551195481 (525h42m57.422278516s) [upload=461143]: Upload finished. err: <nil> Duration: 30.003088155s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X...........X.X......X..............X..X.X...X.XX..XX.X.....] duration: 3.702764271s trace: https://jaeger.dev.storj.io/trace/610bf000d9f46fe4 FAIL (2025-12-27 13:11:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.7027899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.............................X] duration: 1m0.002130745s trace: https://jaeger.dev.storj.io/trace/15e7e71b1e4a765b FAIL (2025-12-27 13:11:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: TPMTEEA74KLTJ5V6YIMXCNHVM7CMXNAPI5JDH53LM3MBII42BGEA) context deadline exceeded; piecestore: (Node ID: 1MT5Nk9DRHLLfHi5sJnzWEjuL3aV2HnbS9Xebdw8vtBMjQmFnV, Piece ID: TZMEUIWTKW7STBEA6VFITVALLVXHIED6C7JVPUYL73CSC2AG5YSA) context deadline exceeded; piecestore: (Node ID: 12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, Piece ID: QOZBFTVMOQ3CAF7BLJVABEIVKIKGXXW6QNLYN27CGTMGG5K2N2BQ) context deadline exceeded ======LOGS====== oise=true , address=77.90.31.13:4651 , piece_id=YWKS2JTDS5QYYPTEEOF54N7LKZGCA6UXQT2DNUA5KRZUXSJ73CZQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.709860762 (525h41m45.580943797s) [noise=true , address=64.49.13.170:4724 , piece_id=FRNFGNNL57GYHDY2EB2DXRQEHUZQTKYYIS3475N7R54H5HATCXVQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.710306931 (525h41m45.581389976s) [noise=true , address=145.79.227.34:4911 , piece_id=M3ODVIPMMESPK5PJTT2N7TX2N6WIZAMKH4U7HMEOVHDPQMWIS5ZA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.710677497 (525h41m45.581760532s) [noise=true , address=77.90.31.12:4624 , piece_id=XUYFLD7JCKYE2LQ66PVOOSRJV4NUZI3VQN6CMG6AUQJETYYN5BCA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.711052121 (525h41m45.582135156s) [noise=true , address=77.90.31.19:4863 , piece_id=6ZLPLFQDENXYKH5BPW3IAMJY7QIDI2SUIRARCBYKNDMOE5BLGTMA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.711430001 (525h41m45.582513036s) [noise=true , address=145.79.227.16:4932 , piece_id=425NEIP6DT3OBKARRSWY76MYMSAZ6VYUWDEY2TKQDJM66F6DQ2YQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.711825103 (525h41m45.582908138s) [noise=true , address=77.90.31.18:4833 , piece_id=TPMTEEA74KLTJ5V6YIMXCNHVM7CMXNAPI5JDH53LM3MBII42BGEA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.712248579 (525h41m45.583331624s) [noise=true , address=145.79.227.40:4078 , piece_id=5JTMTHVDQIR5XO7EBLLBQVVTYB47SQSEIL6F6Y7AZWNH2PKPOYYQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.712670562 (525h41m45.583753607s) [noise=true , address=77.90.31.11:4565 , piece_id=N56S54JPLASZYOGS6VX6YK2ED6GWJDIKCI6TZPGIEFSA4LXEAZJA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.713324337 (525h41m45.584407372s) [noise=true , address=64.49.13.169:4707 , piece_id=CMVRTRWEMPOFKBA6DJ7TOVSCM56IOGVDNKP327EGO2CREAYWF6WA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.713941212 (525h41m45.585024247s) [noise=true , address=145.79.227.43:4162 , piece_id=5D3REI4QA47RUUN3IMIOW6CXFZZAZRYLKVAZCL6YR4MPG7UV4KPQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.715054432 (525h41m45.586137467s) [noise=true , address=77.90.31.15:4718 , piece_id=IHG7SIDROBAPTLWNBEXSIZYG5L4LHI7N4CEPY3APQZMEQJFRACRQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.715571597 (525h41m45.586654632s) [noise=true , address=45.91.116.7:4020 , piece_id=TZMEUIWTKW7STBEA6VFITVALLVXHIED6C7JVPUYL73CSC2AG5YSA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.716503260 (525h41m45.587586305s) [noise=true , address=77.90.31.21:4907 , piece_id=KV4WOT4TC5QJEUEXWGZ6TPHMYALHLB3ZEOW35RZVFTCMGJVWH6ZA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.717147277 (525h41m45.588230312s) [noise=true , address=145.79.227.30:4796 , piece_id=ISVTXMY2AMJI3RDRURII7CX4R7PVPCW7ES7TZFMEV4TXPUU4RYVA, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:45.718560367 (525h41m45.589643402s) [noise=true , address=77.90.31.17:4792 , piece_id=QOZBFTVMOQ3CAF7BLJVABEIVKIKGXXW6QNLYN27CGTMGG5K2N2BQ, seg_pos={0 0}, upload=461126]: Uploading piece... Dec 27 13:10:47.060058932 (525h41m46.931141967s) [upload=461126]: Next returned nil segment Dec 27 13:10:47.060073020 (525h41m46.931156065s) [upload=461126]: Waiting for error group managing segments... Dec 27 13:10:47.409551807 (525h41m47.280634842s) [noise=true , address=64.49.13.170:4724 , piece_id=FRNFGNNL57GYHDY2EB2DXRQEHUZQTKYYIS3475N7R54H5HATCXVQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.599519939 (525h41m47.470602974s) [noise=true , address=145.79.227.16:4932 , piece_id=425NEIP6DT3OBKARRSWY76MYMSAZ6VYUWDEY2TKQDJM66F6DQ2YQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.605459099 (525h41m47.476542144s) [noise=true , address=145.79.227.43:4162 , piece_id=5D3REI4QA47RUUN3IMIOW6CXFZZAZRYLKVAZCL6YR4MPG7UV4KPQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.707001222 (525h41m47.578084257s) [noise=true , address=145.79.227.34:4911 , piece_id=M3ODVIPMMESPK5PJTT2N7TX2N6WIZAMKH4U7HMEOVHDPQMWIS5ZA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.820931805 (525h41m47.69201485s) [noise=true , address=77.90.31.18:4833 , piece_id=TPMTEEA74KLTJ5V6YIMXCNHVM7CMXNAPI5JDH53LM3MBII42BGEA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.824299398 (525h41m47.695382433s) [noise=true , address=45.91.116.7:4020 , piece_id=TZMEUIWTKW7STBEA6VFITVALLVXHIED6C7JVPUYL73CSC2AG5YSA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.825288943 (525h41m47.696371978s) [noise=true , address=77.90.31.17:4792 , piece_id=QOZBFTVMOQ3CAF7BLJVABEIVKIKGXXW6QNLYN27CGTMGG5K2N2BQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:47.852715927 (525h41m47.723798972s) [noise=true , address=77.90.31.19:4863 , piece_id=6ZLPLFQDENXYKH5BPW3IAMJY7QIDI2SUIRARCBYKNDMOE5BLGTMA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.008917121 (525h41m47.880000166s) [noise=true , address=77.90.31.21:4907 , piece_id=KV4WOT4TC5QJEUEXWGZ6TPHMYALHLB3ZEOW35RZVFTCMGJVWH6ZA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.038473493 (525h41m47.909556538s) [noise=true , address=77.90.31.12:4624 , piece_id=XUYFLD7JCKYE2LQ66PVOOSRJV4NUZI3VQN6CMG6AUQJETYYN5BCA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.505751901 (525h41m48.376834936s) [noise=true , address=145.79.227.44:4228 , piece_id=TVURXWI4IKDIXJ34HYZ5D4B4FGR6U4FUCZXACIV2GNGDPO4SAOLQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.521156504 (525h41m48.392239539s) [noise=true , address=145.79.227.30:4796 , piece_id=ISVTXMY2AMJI3RDRURII7CX4R7PVPCW7ES7TZFMEV4TXPUU4RYVA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.545623223 (525h41m48.416706268s) [noise=true , address=77.90.31.16:4745 , piece_id=7K2OUZKFBFNXQMDWDNJHYAA3LIHCIA2EL7FLKMVKOL744V4JN6GQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.552502362 (525h41m48.423585397s) [noise=true , address=145.79.227.40:4078 , piece_id=5JTMTHVDQIR5XO7EBLLBQVVTYB47SQSEIL6F6Y7AZWNH2PKPOYYQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: <nil> Dec 27 13:10:48.552521197 (525h41m48.423604232s) [seg_pos={0 0}, upload=461126]: Segment reached optimal threshold of 14 pieces. Dec 27 13:10:48.552693505 (525h41m48.423776551s) [noise=true , address=64.49.13.169:4707 , piece_id=CMVRTRWEMPOFKBA6DJ7TOVSCM56IOGVDNKP327EGO2CREAYWF6WA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:43628->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:43628->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43628->64.49.13.169:4707: use of closed network connection Dec 27 13:10:48.552701241 (525h41m48.423784276s) [noise=true , address=64.49.13.169:4707 , piece_id=CMVRTRWEMPOFKBA6DJ7TOVSCM56IOGVDNKP327EGO2CREAYWF6WA, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553068299 (525h41m48.424151344s) [noise=true , address=77.90.31.15:4718 , piece_id=IHG7SIDROBAPTLWNBEXSIZYG5L4LHI7N4CEPY3APQZMEQJFRACRQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: expected piece hash; context canceled; EOF Dec 27 13:10:48.553078509 (525h41m48.424161554s) [noise=true , address=77.90.31.15:4718 , piece_id=IHG7SIDROBAPTLWNBEXSIZYG5L4LHI7N4CEPY3APQZMEQJFRACRQ, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553264022 (525h41m48.424347068s) [noise=true , address=77.90.31.13:4651 , piece_id=YWKS2JTDS5QYYPTEEOF54N7LKZGCA6UXQT2DNUA5KRZUXSJ73CZQ, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 27 13:10:48.553272158 (525h41m48.424355193s) [noise=true , address=77.90.31.13:4651 , piece_id=YWKS2JTDS5QYYPTEEOF54N7LKZGCA6UXQT2DNUA5KRZUXSJ73CZQ, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553429377 (525h41m48.424512412s) [noise=true , address=77.90.31.11:4565 , piece_id=N56S54JPLASZYOGS6VX6YK2ED6GWJDIKCI6TZPGIEFSA4LXEAZJA, seg_pos={0 0}, upload=461126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: expected piece hash; context canceled; EOF Dec 27 13:10:48.553437763 (525h41m48.424520798s) [noise=true , address=77.90.31.11:4565 , piece_id=N56S54JPLASZYOGS6VX6YK2ED6GWJDIKCI6TZPGIEFSA4LXEAZJA, seg_pos={0 0}, upload=461126]: Long tail cancellation due to optimal threshold Dec 27 13:10:48.553456298 (525h41m48.424539343s) [seg_pos={0 0}, upload=461126]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 13:10:48.553546450 (525h41m48.424629495s) [upload=461126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:10:48.553553333 (525h41m48.424636388s) [upload=461126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:10:48.860753135 (525h41m48.73183618s) [upload=461126]: Done uploading segments... Dec 27 13:10:48.860772321 (525h41m48.731855356s) [upload=461126]: Upload finished. err: <nil> Duration: 1m0.002152235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX...X..X.X..XX.X......X..X..XXX...........XX..X...XX......] duration: 5.894649148s trace: https://jaeger.dev.storj.io/trace/7ee7825713d0a730 FAIL (2025-12-27 13:11:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 13:06:54.898971928 (525h37m54.770054973s) [upload=461070]: Starting upload Dec 27 13:06:54.899120570 (525h37m54.770203616s) [upload=461070]: Done starting upload Dec 27 13:06:54.899215852 (525h37m54.770298887s) [upload=461070]: Uploading segments... Dec 27 13:06:54.899380366 (525h37m54.770463411s) [upload=461070]: Got next segment. Inline: false Dec 27 13:06:54.899415372 (525h37m54.770498417s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:06:54.899439468 (525h37m54.770522503s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:06:55.151045269 (525h37m55.022128314s) [seg_pos={0 0}, upload=461070]: Begin upload segment... Dec 27 13:06:55.151104031 (525h37m55.022187076s) [seg_pos={0 0}, upload=461070]: Done begin upload segment. Dec 27 13:06:55.151138627 (525h37m55.022221682s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.151247985 (525h37m55.02233105s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152076713 (525h37m55.023159758s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152404376 (525h37m55.023487422s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152753742 (525h37m55.023836787s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153091465 (525h37m55.02417451s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153336832 (525h37m55.024419887s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153646452 (525h37m55.024729497s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154032517 (525h37m55.025115552s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154493865 (525h37m55.02557691s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154833071 (525h37m55.025916116s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155171837 (525h37m55.026254882s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155529477 (525h37m55.026612513s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155810122 (525h37m55.026893157s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155992739 (525h37m55.027075774s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.156658727 (525h37m55.027741772s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:56.866645838 (525h37m56.737728873s) [upload=461070]: Next returned nil segment Dec 27 13:06:56.866658561 (525h37m56.737741596s) [upload=461070]: Waiting for error group managing segments... Dec 27 13:06:57.149778482 (525h37m57.020861517s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.159681608 (525h37m57.030764643s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.160196178 (525h37m57.031279213s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161667410 (525h37m57.032750455s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161792949 (525h37m57.032875984s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.163816041 (525h37m57.034899076s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.167344960 (525h37m57.038428006s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.172138068 (525h37m57.043221103s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.182226408 (525h37m57.053309453s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.189580451 (525h37m57.060663486s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.194174238 (525h37m57.065257273s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.353179021 (525h37m57.224262066s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447756223 (525h37m57.318839268s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447780087 (525h37m57.318863132s) [seg_pos={0 0}, upload=461070]: Segment reached optimal threshold of 13 pieces. Dec 27 13:06:57.448060742 (525h37m57.319143787s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448070942 (525h37m57.319153987s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448122259 (525h37m57.319205294s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448138501 (525h37m57.319221536s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448332379 (525h37m57.319415414s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448339072 (525h37m57.319422107s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448356616 (525h37m57.319439651s) [seg_pos={0 0}, upload=461070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:06:57.448434664 (525h37m57.319517709s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:06:57.448439683 (525h37m57.319522739s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:06:57.746269563 (525h37m57.617352598s) [upload=461070]: Done uploading segments... Dec 27 13:06:57.746291474 (525h37m57.617374509s) [upload=461070]: Upload finished. err: <nil> Duration: 5.89467684s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X...........X.X......X..............X..X.X...X.XX..XX.X....] duration: 5.001680705s trace: https://jaeger.dev.storj.io/trace/610bf000d9f46fe4 FAIL (2025-12-27 13:11:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001748633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 39.643019405s trace: https://jaeger.dev.storj.io/trace/67980cd8fff84f06 FAIL (2025-12-27 13:10:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sy3iQ81wRjoqXs8RhuZW677SBDn4VP6tw14hbozoWenFfgdqu, Piece ID: 5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA) context deadline exceeded; piecestore: (Node ID: 1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, Piece ID: 5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA) context deadline exceeded; piecestore: (Node ID: 124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, Piece ID: KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA) context deadline exceeded ======LOGS====== 13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461479694 (525h13m42.332562739s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461737203 (525h13m42.332820239s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462003020 (525h13m42.333086055s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462034240 (525h13m42.333117285s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462525545 (525h13m42.33360859s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462739873 (525h13m42.333822918s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462797022 (525h13m42.333880057s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463073168 (525h13m42.334156203s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463329756 (525h13m42.334412801s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463569683 (525h13m42.334652728s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463635078 (525h13m42.334718123s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463923096 (525h13m42.335006161s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.464270568 (525h13m42.335353603s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.467068917 (525h13m42.338151962s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:43.771082828 (525h13m43.642165873s) [upload=460711]: Next returned nil segment Dec 27 12:42:43.771096956 (525h13m43.642179991s) [upload=460711]: Waiting for error group managing segments... Dec 27 12:42:44.300665304 (525h13m44.171748339s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.334817064 (525h13m44.205900099s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.357754701 (525h13m44.228837736s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.370947372 (525h13m44.242030417s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.544213540 (525h13m44.415296585s) [noise=true , address=77.90.31.13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.648013103 (525h13m44.519096138s) [noise=true , address=77.90.31.11:4570 , piece_id=6BP3FAA6SZ7CMRBV3DWLDTXPE6IZ4ZD5T6CPQZ2UEKVCU4SFNF5A, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.701365715 (525h13m44.57244876s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.731017397 (525h13m44.602100442s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.184136271 (525h13m45.055219306s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.305192633 (525h13m45.176275678s) [noise=true , address=77.90.31.19:4845 , piece_id=5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.331752207 (525h13m45.202835252s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.337387216 (525h13m45.208470251s) [noise=true , address=45.91.116.13:4164 , piece_id=5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.400681550 (525h13m45.271764595s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.548984854 (525h13m45.420067899s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.549007277 (525h13m45.420090312s) [seg_pos={0 0}, upload=460711]: Segment reached optimal threshold of 14 pieces. Dec 27 12:42:45.549218289 (525h13m45.420301334s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549228118 (525h13m45.420311153s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549354449 (525h13m45.420437494s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14ZdgenD4m2bMMmaHVyK3YivrzdSnrp72JvMJsk1uecW6KKFjn): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549418640 (525h13m45.420501685s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549636385 (525h13m45.42071942s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection Dec 27 12:42:45.549643619 (525h13m45.420726664s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549904335 (525h13m45.42098738s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection Dec 27 12:42:45.549913032 (525h13m45.420996067s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549933541 (525h13m45.421016576s) [seg_pos={0 0}, upload=460711]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:42:45.550005027 (525h13m45.421088062s) [upload=460711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:42:45.550009655 (525h13m45.42109269s) [upload=460711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:42:45.852172915 (525h13m45.72325596s) [upload=460711]: Done uploading segments... Dec 27 12:42:45.852195298 (525h13m45.723278343s) [upload=460711]: Upload finished. err: <nil> Duration: 39.643039274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX...X..X.X..XX.X......X..X..XXX...........XX..X...XX.....] duration: 5.931353908s trace: https://jaeger.dev.storj.io/trace/7ee7825713d0a730 FAIL (2025-12-27 13:10:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 13:06:54.898971928 (525h37m54.770054973s) [upload=461070]: Starting upload Dec 27 13:06:54.899120570 (525h37m54.770203616s) [upload=461070]: Done starting upload Dec 27 13:06:54.899215852 (525h37m54.770298887s) [upload=461070]: Uploading segments... Dec 27 13:06:54.899380366 (525h37m54.770463411s) [upload=461070]: Got next segment. Inline: false Dec 27 13:06:54.899415372 (525h37m54.770498417s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:06:54.899439468 (525h37m54.770522503s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:06:55.151045269 (525h37m55.022128314s) [seg_pos={0 0}, upload=461070]: Begin upload segment... Dec 27 13:06:55.151104031 (525h37m55.022187076s) [seg_pos={0 0}, upload=461070]: Done begin upload segment. Dec 27 13:06:55.151138627 (525h37m55.022221682s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.151247985 (525h37m55.02233105s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152076713 (525h37m55.023159758s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152404376 (525h37m55.023487422s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152753742 (525h37m55.023836787s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153091465 (525h37m55.02417451s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153336832 (525h37m55.024419887s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153646452 (525h37m55.024729497s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154032517 (525h37m55.025115552s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154493865 (525h37m55.02557691s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154833071 (525h37m55.025916116s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155171837 (525h37m55.026254882s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155529477 (525h37m55.026612513s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155810122 (525h37m55.026893157s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155992739 (525h37m55.027075774s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.156658727 (525h37m55.027741772s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:56.866645838 (525h37m56.737728873s) [upload=461070]: Next returned nil segment Dec 27 13:06:56.866658561 (525h37m56.737741596s) [upload=461070]: Waiting for error group managing segments... Dec 27 13:06:57.149778482 (525h37m57.020861517s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.159681608 (525h37m57.030764643s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.160196178 (525h37m57.031279213s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161667410 (525h37m57.032750455s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161792949 (525h37m57.032875984s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.163816041 (525h37m57.034899076s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.167344960 (525h37m57.038428006s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.172138068 (525h37m57.043221103s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.182226408 (525h37m57.053309453s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.189580451 (525h37m57.060663486s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.194174238 (525h37m57.065257273s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.353179021 (525h37m57.224262066s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447756223 (525h37m57.318839268s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447780087 (525h37m57.318863132s) [seg_pos={0 0}, upload=461070]: Segment reached optimal threshold of 13 pieces. Dec 27 13:06:57.448060742 (525h37m57.319143787s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448070942 (525h37m57.319153987s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448122259 (525h37m57.319205294s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448138501 (525h37m57.319221536s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448332379 (525h37m57.319415414s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448339072 (525h37m57.319422107s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448356616 (525h37m57.319439651s) [seg_pos={0 0}, upload=461070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:06:57.448434664 (525h37m57.319517709s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:06:57.448439683 (525h37m57.319522739s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:06:57.746269563 (525h37m57.617352598s) [upload=461070]: Done uploading segments... Dec 27 13:06:57.746291474 (525h37m57.617374509s) [upload=461070]: Upload finished. err: <nil> Duration: 5.931381811s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..X...........X.X......X..............X..X.X...X.XX..XX.X...] duration: 3.449929851s trace: https://jaeger.dev.storj.io/trace/610bf000d9f46fe4 FAIL (2025-12-27 13:10:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.449967032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 40.887731883s trace: https://jaeger.dev.storj.io/trace/67980cd8fff84f06 FAIL (2025-12-27 13:09:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sy3iQ81wRjoqXs8RhuZW677SBDn4VP6tw14hbozoWenFfgdqu, Piece ID: 5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA) context deadline exceeded; piecestore: (Node ID: 1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, Piece ID: 5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA) context deadline exceeded; piecestore: (Node ID: 124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, Piece ID: KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA) context deadline exceeded ======LOGS====== 13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461479694 (525h13m42.332562739s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461737203 (525h13m42.332820239s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462003020 (525h13m42.333086055s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462034240 (525h13m42.333117285s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462525545 (525h13m42.33360859s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462739873 (525h13m42.333822918s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462797022 (525h13m42.333880057s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463073168 (525h13m42.334156203s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463329756 (525h13m42.334412801s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463569683 (525h13m42.334652728s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463635078 (525h13m42.334718123s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463923096 (525h13m42.335006161s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.464270568 (525h13m42.335353603s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.467068917 (525h13m42.338151962s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:43.771082828 (525h13m43.642165873s) [upload=460711]: Next returned nil segment Dec 27 12:42:43.771096956 (525h13m43.642179991s) [upload=460711]: Waiting for error group managing segments... Dec 27 12:42:44.300665304 (525h13m44.171748339s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.334817064 (525h13m44.205900099s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.357754701 (525h13m44.228837736s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.370947372 (525h13m44.242030417s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.544213540 (525h13m44.415296585s) [noise=true , address=77.90.31.13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.648013103 (525h13m44.519096138s) [noise=true , address=77.90.31.11:4570 , piece_id=6BP3FAA6SZ7CMRBV3DWLDTXPE6IZ4ZD5T6CPQZ2UEKVCU4SFNF5A, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.701365715 (525h13m44.57244876s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.731017397 (525h13m44.602100442s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.184136271 (525h13m45.055219306s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.305192633 (525h13m45.176275678s) [noise=true , address=77.90.31.19:4845 , piece_id=5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.331752207 (525h13m45.202835252s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.337387216 (525h13m45.208470251s) [noise=true , address=45.91.116.13:4164 , piece_id=5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.400681550 (525h13m45.271764595s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.548984854 (525h13m45.420067899s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.549007277 (525h13m45.420090312s) [seg_pos={0 0}, upload=460711]: Segment reached optimal threshold of 14 pieces. Dec 27 12:42:45.549218289 (525h13m45.420301334s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549228118 (525h13m45.420311153s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549354449 (525h13m45.420437494s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14ZdgenD4m2bMMmaHVyK3YivrzdSnrp72JvMJsk1uecW6KKFjn): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549418640 (525h13m45.420501685s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549636385 (525h13m45.42071942s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection Dec 27 12:42:45.549643619 (525h13m45.420726664s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549904335 (525h13m45.42098738s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection Dec 27 12:42:45.549913032 (525h13m45.420996067s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549933541 (525h13m45.421016576s) [seg_pos={0 0}, upload=460711]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:42:45.550005027 (525h13m45.421088062s) [upload=460711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:42:45.550009655 (525h13m45.42109269s) [upload=460711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:42:45.852172915 (525h13m45.72325596s) [upload=460711]: Done uploading segments... Dec 27 12:42:45.852195298 (525h13m45.723278343s) [upload=460711]: Upload finished. err: <nil> Duration: 40.887750319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..XX...X..X.X..XX.X......X..X..XXX...........XX..X...XX....] duration: 5.657984767s trace: https://jaeger.dev.storj.io/trace/7ee7825713d0a730 FAIL (2025-12-27 13:09:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 13:06:54.898971928 (525h37m54.770054973s) [upload=461070]: Starting upload Dec 27 13:06:54.899120570 (525h37m54.770203616s) [upload=461070]: Done starting upload Dec 27 13:06:54.899215852 (525h37m54.770298887s) [upload=461070]: Uploading segments... Dec 27 13:06:54.899380366 (525h37m54.770463411s) [upload=461070]: Got next segment. Inline: false Dec 27 13:06:54.899415372 (525h37m54.770498417s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:06:54.899439468 (525h37m54.770522503s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:06:55.151045269 (525h37m55.022128314s) [seg_pos={0 0}, upload=461070]: Begin upload segment... Dec 27 13:06:55.151104031 (525h37m55.022187076s) [seg_pos={0 0}, upload=461070]: Done begin upload segment. Dec 27 13:06:55.151138627 (525h37m55.022221682s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.151247985 (525h37m55.02233105s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152076713 (525h37m55.023159758s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152404376 (525h37m55.023487422s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152753742 (525h37m55.023836787s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153091465 (525h37m55.02417451s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153336832 (525h37m55.024419887s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153646452 (525h37m55.024729497s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154032517 (525h37m55.025115552s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154493865 (525h37m55.02557691s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154833071 (525h37m55.025916116s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155171837 (525h37m55.026254882s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155529477 (525h37m55.026612513s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155810122 (525h37m55.026893157s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155992739 (525h37m55.027075774s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.156658727 (525h37m55.027741772s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:56.866645838 (525h37m56.737728873s) [upload=461070]: Next returned nil segment Dec 27 13:06:56.866658561 (525h37m56.737741596s) [upload=461070]: Waiting for error group managing segments... Dec 27 13:06:57.149778482 (525h37m57.020861517s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.159681608 (525h37m57.030764643s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.160196178 (525h37m57.031279213s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161667410 (525h37m57.032750455s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161792949 (525h37m57.032875984s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.163816041 (525h37m57.034899076s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.167344960 (525h37m57.038428006s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.172138068 (525h37m57.043221103s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.182226408 (525h37m57.053309453s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.189580451 (525h37m57.060663486s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.194174238 (525h37m57.065257273s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.353179021 (525h37m57.224262066s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447756223 (525h37m57.318839268s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447780087 (525h37m57.318863132s) [seg_pos={0 0}, upload=461070]: Segment reached optimal threshold of 13 pieces. Dec 27 13:06:57.448060742 (525h37m57.319143787s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448070942 (525h37m57.319153987s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448122259 (525h37m57.319205294s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448138501 (525h37m57.319221536s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448332379 (525h37m57.319415414s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448339072 (525h37m57.319422107s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448356616 (525h37m57.319439651s) [seg_pos={0 0}, upload=461070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:06:57.448434664 (525h37m57.319517709s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:06:57.448439683 (525h37m57.319522739s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:06:57.746269563 (525h37m57.617352598s) [upload=461070]: Done uploading segments... Dec 27 13:06:57.746291474 (525h37m57.617374509s) [upload=461070]: Upload finished. err: <nil> Duration: 5.658003914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 43.754250322s trace: https://jaeger.dev.storj.io/trace/67980cd8fff84f06 FAIL (2025-12-27 13:09:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sy3iQ81wRjoqXs8RhuZW677SBDn4VP6tw14hbozoWenFfgdqu, Piece ID: 5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA) context deadline exceeded; piecestore: (Node ID: 1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, Piece ID: 5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA) context deadline exceeded; piecestore: (Node ID: 124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, Piece ID: KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA) context deadline exceeded ======LOGS====== 13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461479694 (525h13m42.332562739s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461737203 (525h13m42.332820239s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462003020 (525h13m42.333086055s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462034240 (525h13m42.333117285s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462525545 (525h13m42.33360859s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462739873 (525h13m42.333822918s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462797022 (525h13m42.333880057s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463073168 (525h13m42.334156203s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463329756 (525h13m42.334412801s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463569683 (525h13m42.334652728s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463635078 (525h13m42.334718123s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463923096 (525h13m42.335006161s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.464270568 (525h13m42.335353603s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.467068917 (525h13m42.338151962s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:43.771082828 (525h13m43.642165873s) [upload=460711]: Next returned nil segment Dec 27 12:42:43.771096956 (525h13m43.642179991s) [upload=460711]: Waiting for error group managing segments... Dec 27 12:42:44.300665304 (525h13m44.171748339s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.334817064 (525h13m44.205900099s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.357754701 (525h13m44.228837736s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.370947372 (525h13m44.242030417s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.544213540 (525h13m44.415296585s) [noise=true , address=77.90.31.13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.648013103 (525h13m44.519096138s) [noise=true , address=77.90.31.11:4570 , piece_id=6BP3FAA6SZ7CMRBV3DWLDTXPE6IZ4ZD5T6CPQZ2UEKVCU4SFNF5A, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.701365715 (525h13m44.57244876s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.731017397 (525h13m44.602100442s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.184136271 (525h13m45.055219306s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.305192633 (525h13m45.176275678s) [noise=true , address=77.90.31.19:4845 , piece_id=5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.331752207 (525h13m45.202835252s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.337387216 (525h13m45.208470251s) [noise=true , address=45.91.116.13:4164 , piece_id=5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.400681550 (525h13m45.271764595s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.548984854 (525h13m45.420067899s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.549007277 (525h13m45.420090312s) [seg_pos={0 0}, upload=460711]: Segment reached optimal threshold of 14 pieces. Dec 27 12:42:45.549218289 (525h13m45.420301334s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549228118 (525h13m45.420311153s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549354449 (525h13m45.420437494s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14ZdgenD4m2bMMmaHVyK3YivrzdSnrp72JvMJsk1uecW6KKFjn): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549418640 (525h13m45.420501685s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549636385 (525h13m45.42071942s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection Dec 27 12:42:45.549643619 (525h13m45.420726664s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549904335 (525h13m45.42098738s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection Dec 27 12:42:45.549913032 (525h13m45.420996067s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549933541 (525h13m45.421016576s) [seg_pos={0 0}, upload=460711]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:42:45.550005027 (525h13m45.421088062s) [upload=460711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:42:45.550009655 (525h13m45.42109269s) [upload=460711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:42:45.852172915 (525h13m45.72325596s) [upload=460711]: Done uploading segments... Dec 27 12:42:45.852195298 (525h13m45.723278343s) [upload=460711]: Upload finished. err: <nil> Duration: 43.754279448s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...X...........X.X......X..............X..X.X...X.XX..XX.X..] duration: 4.262425263s trace: https://jaeger.dev.storj.io/trace/610bf000d9f46fe4 FAIL (2025-12-27 13:09:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.262457804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX..XX...X..X.X..XX.X......X..X..XXX...........XX..X...XX...] duration: 5.90798356s trace: https://jaeger.dev.storj.io/trace/7ee7825713d0a730 FAIL (2025-12-27 13:09:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 13:06:54.898971928 (525h37m54.770054973s) [upload=461070]: Starting upload Dec 27 13:06:54.899120570 (525h37m54.770203616s) [upload=461070]: Done starting upload Dec 27 13:06:54.899215852 (525h37m54.770298887s) [upload=461070]: Uploading segments... Dec 27 13:06:54.899380366 (525h37m54.770463411s) [upload=461070]: Got next segment. Inline: false Dec 27 13:06:54.899415372 (525h37m54.770498417s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:06:54.899439468 (525h37m54.770522503s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:06:55.151045269 (525h37m55.022128314s) [seg_pos={0 0}, upload=461070]: Begin upload segment... Dec 27 13:06:55.151104031 (525h37m55.022187076s) [seg_pos={0 0}, upload=461070]: Done begin upload segment. Dec 27 13:06:55.151138627 (525h37m55.022221682s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.151247985 (525h37m55.02233105s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152076713 (525h37m55.023159758s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152404376 (525h37m55.023487422s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152753742 (525h37m55.023836787s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153091465 (525h37m55.02417451s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153336832 (525h37m55.024419887s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153646452 (525h37m55.024729497s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154032517 (525h37m55.025115552s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154493865 (525h37m55.02557691s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154833071 (525h37m55.025916116s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155171837 (525h37m55.026254882s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155529477 (525h37m55.026612513s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155810122 (525h37m55.026893157s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155992739 (525h37m55.027075774s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.156658727 (525h37m55.027741772s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:56.866645838 (525h37m56.737728873s) [upload=461070]: Next returned nil segment Dec 27 13:06:56.866658561 (525h37m56.737741596s) [upload=461070]: Waiting for error group managing segments... Dec 27 13:06:57.149778482 (525h37m57.020861517s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.159681608 (525h37m57.030764643s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.160196178 (525h37m57.031279213s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161667410 (525h37m57.032750455s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161792949 (525h37m57.032875984s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.163816041 (525h37m57.034899076s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.167344960 (525h37m57.038428006s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.172138068 (525h37m57.043221103s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.182226408 (525h37m57.053309453s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.189580451 (525h37m57.060663486s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.194174238 (525h37m57.065257273s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.353179021 (525h37m57.224262066s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447756223 (525h37m57.318839268s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447780087 (525h37m57.318863132s) [seg_pos={0 0}, upload=461070]: Segment reached optimal threshold of 13 pieces. Dec 27 13:06:57.448060742 (525h37m57.319143787s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448070942 (525h37m57.319153987s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448122259 (525h37m57.319205294s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448138501 (525h37m57.319221536s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448332379 (525h37m57.319415414s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448339072 (525h37m57.319422107s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448356616 (525h37m57.319439651s) [seg_pos={0 0}, upload=461070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:06:57.448434664 (525h37m57.319517709s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:06:57.448439683 (525h37m57.319522739s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:06:57.746269563 (525h37m57.617352598s) [upload=461070]: Done uploading segments... Dec 27 13:06:57.746291474 (525h37m57.617374509s) [upload=461070]: Upload finished. err: <nil> Duration: 5.908018805s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [....X...........X.X......X..............X..X.X...X.XX..XX.X.] duration: 5.001179277s trace: https://jaeger.dev.storj.io/trace/610bf000d9f46fe4 FAIL (2025-12-27 13:08:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001234972s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.....X...........X.X......X..............X..X.X...X.XX..XX.X] duration: 5.000980368s trace: https://jaeger.dev.storj.io/trace/610bf000d9f46fe4 FAIL (2025-12-27 13:08:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00100209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX..XX...X..X.X..XX.X......X..X..XXX...........XX..X...XX..] duration: 5.996418234s trace: https://jaeger.dev.storj.io/trace/7ee7825713d0a730 FAIL (2025-12-27 13:08:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 13:06:54.898971928 (525h37m54.770054973s) [upload=461070]: Starting upload Dec 27 13:06:54.899120570 (525h37m54.770203616s) [upload=461070]: Done starting upload Dec 27 13:06:54.899215852 (525h37m54.770298887s) [upload=461070]: Uploading segments... Dec 27 13:06:54.899380366 (525h37m54.770463411s) [upload=461070]: Got next segment. Inline: false Dec 27 13:06:54.899415372 (525h37m54.770498417s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:06:54.899439468 (525h37m54.770522503s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:06:55.151045269 (525h37m55.022128314s) [seg_pos={0 0}, upload=461070]: Begin upload segment... Dec 27 13:06:55.151104031 (525h37m55.022187076s) [seg_pos={0 0}, upload=461070]: Done begin upload segment. Dec 27 13:06:55.151138627 (525h37m55.022221682s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.151247985 (525h37m55.02233105s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152076713 (525h37m55.023159758s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152404376 (525h37m55.023487422s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152753742 (525h37m55.023836787s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153091465 (525h37m55.02417451s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153336832 (525h37m55.024419887s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153646452 (525h37m55.024729497s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154032517 (525h37m55.025115552s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154493865 (525h37m55.02557691s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154833071 (525h37m55.025916116s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155171837 (525h37m55.026254882s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155529477 (525h37m55.026612513s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155810122 (525h37m55.026893157s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155992739 (525h37m55.027075774s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.156658727 (525h37m55.027741772s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:56.866645838 (525h37m56.737728873s) [upload=461070]: Next returned nil segment Dec 27 13:06:56.866658561 (525h37m56.737741596s) [upload=461070]: Waiting for error group managing segments... Dec 27 13:06:57.149778482 (525h37m57.020861517s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.159681608 (525h37m57.030764643s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.160196178 (525h37m57.031279213s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161667410 (525h37m57.032750455s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161792949 (525h37m57.032875984s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.163816041 (525h37m57.034899076s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.167344960 (525h37m57.038428006s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.172138068 (525h37m57.043221103s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.182226408 (525h37m57.053309453s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.189580451 (525h37m57.060663486s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.194174238 (525h37m57.065257273s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.353179021 (525h37m57.224262066s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447756223 (525h37m57.318839268s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447780087 (525h37m57.318863132s) [seg_pos={0 0}, upload=461070]: Segment reached optimal threshold of 13 pieces. Dec 27 13:06:57.448060742 (525h37m57.319143787s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448070942 (525h37m57.319153987s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448122259 (525h37m57.319205294s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448138501 (525h37m57.319221536s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448332379 (525h37m57.319415414s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448339072 (525h37m57.319422107s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448356616 (525h37m57.319439651s) [seg_pos={0 0}, upload=461070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:06:57.448434664 (525h37m57.319517709s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:06:57.448439683 (525h37m57.319522739s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:06:57.746269563 (525h37m57.617352598s) [upload=461070]: Done uploading segments... Dec 27 13:06:57.746291474 (525h37m57.617374509s) [upload=461070]: Upload finished. err: <nil> Duration: 5.996448271s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X................................X..........................] duration: 38.460064989s trace: https://jaeger.dev.storj.io/trace/67980cd8fff84f06 FAIL (2025-12-27 13:07:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sy3iQ81wRjoqXs8RhuZW677SBDn4VP6tw14hbozoWenFfgdqu, Piece ID: 5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA) context deadline exceeded; piecestore: (Node ID: 1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, Piece ID: 5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA) context deadline exceeded; piecestore: (Node ID: 124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, Piece ID: KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA) context deadline exceeded ======LOGS====== 13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461479694 (525h13m42.332562739s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461737203 (525h13m42.332820239s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462003020 (525h13m42.333086055s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462034240 (525h13m42.333117285s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462525545 (525h13m42.33360859s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462739873 (525h13m42.333822918s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462797022 (525h13m42.333880057s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463073168 (525h13m42.334156203s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463329756 (525h13m42.334412801s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463569683 (525h13m42.334652728s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463635078 (525h13m42.334718123s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463923096 (525h13m42.335006161s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.464270568 (525h13m42.335353603s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.467068917 (525h13m42.338151962s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:43.771082828 (525h13m43.642165873s) [upload=460711]: Next returned nil segment Dec 27 12:42:43.771096956 (525h13m43.642179991s) [upload=460711]: Waiting for error group managing segments... Dec 27 12:42:44.300665304 (525h13m44.171748339s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.334817064 (525h13m44.205900099s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.357754701 (525h13m44.228837736s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.370947372 (525h13m44.242030417s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.544213540 (525h13m44.415296585s) [noise=true , address=77.90.31.13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.648013103 (525h13m44.519096138s) [noise=true , address=77.90.31.11:4570 , piece_id=6BP3FAA6SZ7CMRBV3DWLDTXPE6IZ4ZD5T6CPQZ2UEKVCU4SFNF5A, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.701365715 (525h13m44.57244876s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.731017397 (525h13m44.602100442s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.184136271 (525h13m45.055219306s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.305192633 (525h13m45.176275678s) [noise=true , address=77.90.31.19:4845 , piece_id=5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.331752207 (525h13m45.202835252s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.337387216 (525h13m45.208470251s) [noise=true , address=45.91.116.13:4164 , piece_id=5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.400681550 (525h13m45.271764595s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.548984854 (525h13m45.420067899s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.549007277 (525h13m45.420090312s) [seg_pos={0 0}, upload=460711]: Segment reached optimal threshold of 14 pieces. Dec 27 12:42:45.549218289 (525h13m45.420301334s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549228118 (525h13m45.420311153s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549354449 (525h13m45.420437494s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14ZdgenD4m2bMMmaHVyK3YivrzdSnrp72JvMJsk1uecW6KKFjn): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549418640 (525h13m45.420501685s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549636385 (525h13m45.42071942s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection Dec 27 12:42:45.549643619 (525h13m45.420726664s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549904335 (525h13m45.42098738s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection Dec 27 12:42:45.549913032 (525h13m45.420996067s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549933541 (525h13m45.421016576s) [seg_pos={0 0}, upload=460711]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:42:45.550005027 (525h13m45.421088062s) [upload=460711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:42:45.550009655 (525h13m45.42109269s) [upload=460711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:42:45.852172915 (525h13m45.72325596s) [upload=460711]: Done uploading segments... Dec 27 12:42:45.852195298 (525h13m45.723278343s) [upload=460711]: Upload finished. err: <nil> Duration: 38.460084365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.XX..XX...X..X.X..XX.X......X..X..XXX...........XX..X...XX.] duration: 5.636723843s trace: https://jaeger.dev.storj.io/trace/7ee7825713d0a730 FAIL (2025-12-27 13:07:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 13:06:54.898971928 (525h37m54.770054973s) [upload=461070]: Starting upload Dec 27 13:06:54.899120570 (525h37m54.770203616s) [upload=461070]: Done starting upload Dec 27 13:06:54.899215852 (525h37m54.770298887s) [upload=461070]: Uploading segments... Dec 27 13:06:54.899380366 (525h37m54.770463411s) [upload=461070]: Got next segment. Inline: false Dec 27 13:06:54.899415372 (525h37m54.770498417s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:06:54.899439468 (525h37m54.770522503s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:06:55.151045269 (525h37m55.022128314s) [seg_pos={0 0}, upload=461070]: Begin upload segment... Dec 27 13:06:55.151104031 (525h37m55.022187076s) [seg_pos={0 0}, upload=461070]: Done begin upload segment. Dec 27 13:06:55.151138627 (525h37m55.022221682s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.151247985 (525h37m55.02233105s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152076713 (525h37m55.023159758s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152404376 (525h37m55.023487422s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152753742 (525h37m55.023836787s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153091465 (525h37m55.02417451s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153336832 (525h37m55.024419887s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153646452 (525h37m55.024729497s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154032517 (525h37m55.025115552s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154493865 (525h37m55.02557691s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154833071 (525h37m55.025916116s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155171837 (525h37m55.026254882s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155529477 (525h37m55.026612513s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155810122 (525h37m55.026893157s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155992739 (525h37m55.027075774s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.156658727 (525h37m55.027741772s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:56.866645838 (525h37m56.737728873s) [upload=461070]: Next returned nil segment Dec 27 13:06:56.866658561 (525h37m56.737741596s) [upload=461070]: Waiting for error group managing segments... Dec 27 13:06:57.149778482 (525h37m57.020861517s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.159681608 (525h37m57.030764643s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.160196178 (525h37m57.031279213s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161667410 (525h37m57.032750455s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161792949 (525h37m57.032875984s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.163816041 (525h37m57.034899076s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.167344960 (525h37m57.038428006s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.172138068 (525h37m57.043221103s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.182226408 (525h37m57.053309453s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.189580451 (525h37m57.060663486s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.194174238 (525h37m57.065257273s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.353179021 (525h37m57.224262066s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447756223 (525h37m57.318839268s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447780087 (525h37m57.318863132s) [seg_pos={0 0}, upload=461070]: Segment reached optimal threshold of 13 pieces. Dec 27 13:06:57.448060742 (525h37m57.319143787s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448070942 (525h37m57.319153987s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448122259 (525h37m57.319205294s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448138501 (525h37m57.319221536s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448332379 (525h37m57.319415414s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448339072 (525h37m57.319422107s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448356616 (525h37m57.319439651s) [seg_pos={0 0}, upload=461070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:06:57.448434664 (525h37m57.319517709s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:06:57.448439683 (525h37m57.319522739s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:06:57.746269563 (525h37m57.617352598s) [upload=461070]: Done uploading segments... Dec 27 13:06:57.746291474 (525h37m57.617374509s) [upload=461070]: Upload finished. err: <nil> Duration: 5.636793586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.XX..XX...X..X.X..XX.X......X..X..XXX...........XX..X...XX] duration: 30.003120772s trace: https://jaeger.dev.storj.io/trace/7ee7825713d0a730 FAIL (2025-12-27 13:07:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 13:06:54.898971928 (525h37m54.770054973s) [upload=461070]: Starting upload Dec 27 13:06:54.899120570 (525h37m54.770203616s) [upload=461070]: Done starting upload Dec 27 13:06:54.899215852 (525h37m54.770298887s) [upload=461070]: Uploading segments... Dec 27 13:06:54.899380366 (525h37m54.770463411s) [upload=461070]: Got next segment. Inline: false Dec 27 13:06:54.899415372 (525h37m54.770498417s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:06:54.899439468 (525h37m54.770522503s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:06:55.151045269 (525h37m55.022128314s) [seg_pos={0 0}, upload=461070]: Begin upload segment... Dec 27 13:06:55.151104031 (525h37m55.022187076s) [seg_pos={0 0}, upload=461070]: Done begin upload segment. Dec 27 13:06:55.151138627 (525h37m55.022221682s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.151247985 (525h37m55.02233105s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152076713 (525h37m55.023159758s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152404376 (525h37m55.023487422s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.152753742 (525h37m55.023836787s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153091465 (525h37m55.02417451s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153336832 (525h37m55.024419887s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.153646452 (525h37m55.024729497s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154032517 (525h37m55.025115552s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154493865 (525h37m55.02557691s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.154833071 (525h37m55.025916116s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155171837 (525h37m55.026254882s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155529477 (525h37m55.026612513s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155810122 (525h37m55.026893157s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.155992739 (525h37m55.027075774s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:55.156658727 (525h37m55.027741772s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Uploading piece... Dec 27 13:06:56.866645838 (525h37m56.737728873s) [upload=461070]: Next returned nil segment Dec 27 13:06:56.866658561 (525h37m56.737741596s) [upload=461070]: Waiting for error group managing segments... Dec 27 13:06:57.149778482 (525h37m57.020861517s) [noise=true , address=145.79.227.2:4114 , piece_id=SL2LGWFHHA36UXSPHILMF37HPM4ZEOVQD3DLS3VGSQ5ARM5HOEHQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.159681608 (525h37m57.030764643s) [noise=true , address=145.79.227.19:4078 , piece_id=6RV4XAT244DYZW6APSSVBXQ4GZIDVK57IXWTFSMRMUZ3T5PXAUNQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.160196178 (525h37m57.031279213s) [noise=true , address=145.79.227.42:4121 , piece_id=B5TVDYA4V7VO6A2GGVRF4H24LUZ23BIP2Z2JIQNN7TFL2DDR7SJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161667410 (525h37m57.032750455s) [noise=true , address=145.79.227.13:4804 , piece_id=VGIJYZXLPNMUOBVIVB2LSVI37E5DWWGGABYYZH342B7RFVHPENEA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.161792949 (525h37m57.032875984s) [noise=true , address=145.79.227.20:4438 , piece_id=MUHVADL544XVE6XNMVBQ4U5DPU7KN56DZ2X45WURV6YLROPEL3EQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.163816041 (525h37m57.034899076s) [noise=true , address=145.79.227.38:4978 , piece_id=B7FT2LV722OPCANM6U3A47IPOSJ25BML5OMYX2P7S3EXUEKYPXRA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.167344960 (525h37m57.038428006s) [noise=true , address=145.79.227.51:4410 , piece_id=OCR32WNN7RVG3WZHJVZQRDVKEUZ6LLW3JNFYF7P5ITZV6LO23ANQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.172138068 (525h37m57.043221103s) [noise=true , address=145.79.227.52:4445 , piece_id=2LOMTHID7EI3ULSBYEGW7OFUOHYXJ5MDWJC7H6QI2YJSDKLJQNUA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.182226408 (525h37m57.053309453s) [noise=true , address=145.79.227.50:4374 , piece_id=KGDN3HS5FRFHH4LLNPMU4BQOSISWXDYC53THGSJU6N74RSYKYPQQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.189580451 (525h37m57.060663486s) [noise=true , address=145.79.227.21:4475 , piece_id=LI77MHY3YKDTCFTQEGASRTYPP23APIBWB2G4PPZJHJVE4UWL3P3A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.194174238 (525h37m57.065257273s) [noise=true , address=145.79.227.25:4617 , piece_id=J2Z33GEFBEQT6M23X35O7JZ6MJABAYF5NPS64JUPCFKDPTRXYOGQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.353179021 (525h37m57.224262066s) [noise=true , address=77.90.31.11:4556 , piece_id=V2AUJKSC547AXOBS4HMUTODLBK7GWBBO57IFC2MD3HLZJXG7XQTA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447756223 (525h37m57.318839268s) [noise=true , address=45.91.116.5:4297 , piece_id=XF75P36X6XOMH5HWXH65SZDLWI2TCZPHXPS5DQF3ESANFK3YJL4A, seg_pos={0 0}, upload=461070]: Done uploading piece. err: <nil> Dec 27 13:06:57.447780087 (525h37m57.318863132s) [seg_pos={0 0}, upload=461070]: Segment reached optimal threshold of 13 pieces. Dec 27 13:06:57.448060742 (525h37m57.319143787s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448070942 (525h37m57.319153987s) [noise=true , address=45.91.116.10:4081 , piece_id=TXJMK6KI4VP75N7ZVZPPGRGGCK2UXB4PUBT4WMVDH4DNG3RGJAOA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448122259 (525h37m57.319205294s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448138501 (525h37m57.319221536s) [noise=true , address=45.91.116.4:4939 , piece_id=LTPWZ2ILQGB4L7UPFYYFNSSMIXG7LHK3NPQTCQEKYBI4J7RVYEQA, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448332379 (525h37m57.319415414s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 27 13:06:57.448339072 (525h37m57.319422107s) [noise=true , address=77.90.31.19:4842 , piece_id=M7QZYA7C3CCEJYIP46RUMBPNGRKAVBDL66QWZXLE4JCK35DAIMJQ, seg_pos={0 0}, upload=461070]: Long tail cancellation due to optimal threshold Dec 27 13:06:57.448356616 (525h37m57.319439651s) [seg_pos={0 0}, upload=461070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:06:57.448434664 (525h37m57.319517709s) [upload=461070]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:06:57.448439683 (525h37m57.319522739s) [upload=461070]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:06:57.746269563 (525h37m57.617352598s) [upload=461070]: Done uploading segments... Dec 27 13:06:57.746291474 (525h37m57.617374509s) [upload=461070]: Upload finished. err: <nil> Duration: 30.00314136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X................................X.........................] duration: 42.666889039s trace: https://jaeger.dev.storj.io/trace/67980cd8fff84f06 FAIL (2025-12-27 13:07:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sy3iQ81wRjoqXs8RhuZW677SBDn4VP6tw14hbozoWenFfgdqu, Piece ID: 5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA) context deadline exceeded; piecestore: (Node ID: 1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, Piece ID: 5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA) context deadline exceeded; piecestore: (Node ID: 124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, Piece ID: KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA) context deadline exceeded ======LOGS====== 13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461479694 (525h13m42.332562739s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461737203 (525h13m42.332820239s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462003020 (525h13m42.333086055s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462034240 (525h13m42.333117285s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462525545 (525h13m42.33360859s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462739873 (525h13m42.333822918s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462797022 (525h13m42.333880057s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463073168 (525h13m42.334156203s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463329756 (525h13m42.334412801s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463569683 (525h13m42.334652728s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463635078 (525h13m42.334718123s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463923096 (525h13m42.335006161s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.464270568 (525h13m42.335353603s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.467068917 (525h13m42.338151962s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:43.771082828 (525h13m43.642165873s) [upload=460711]: Next returned nil segment Dec 27 12:42:43.771096956 (525h13m43.642179991s) [upload=460711]: Waiting for error group managing segments... Dec 27 12:42:44.300665304 (525h13m44.171748339s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.334817064 (525h13m44.205900099s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.357754701 (525h13m44.228837736s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.370947372 (525h13m44.242030417s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.544213540 (525h13m44.415296585s) [noise=true , address=77.90.31.13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.648013103 (525h13m44.519096138s) [noise=true , address=77.90.31.11:4570 , piece_id=6BP3FAA6SZ7CMRBV3DWLDTXPE6IZ4ZD5T6CPQZ2UEKVCU4SFNF5A, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.701365715 (525h13m44.57244876s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.731017397 (525h13m44.602100442s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.184136271 (525h13m45.055219306s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.305192633 (525h13m45.176275678s) [noise=true , address=77.90.31.19:4845 , piece_id=5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.331752207 (525h13m45.202835252s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.337387216 (525h13m45.208470251s) [noise=true , address=45.91.116.13:4164 , piece_id=5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.400681550 (525h13m45.271764595s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.548984854 (525h13m45.420067899s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.549007277 (525h13m45.420090312s) [seg_pos={0 0}, upload=460711]: Segment reached optimal threshold of 14 pieces. Dec 27 12:42:45.549218289 (525h13m45.420301334s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549228118 (525h13m45.420311153s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549354449 (525h13m45.420437494s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14ZdgenD4m2bMMmaHVyK3YivrzdSnrp72JvMJsk1uecW6KKFjn): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549418640 (525h13m45.420501685s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549636385 (525h13m45.42071942s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection Dec 27 12:42:45.549643619 (525h13m45.420726664s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549904335 (525h13m45.42098738s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection Dec 27 12:42:45.549913032 (525h13m45.420996067s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549933541 (525h13m45.421016576s) [seg_pos={0 0}, upload=460711]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:42:45.550005027 (525h13m45.421088062s) [upload=460711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:42:45.550009655 (525h13m45.42109269s) [upload=460711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:42:45.852172915 (525h13m45.72325596s) [upload=460711]: Done uploading segments... Dec 27 12:42:45.852195298 (525h13m45.723278343s) [upload=460711]: Upload finished. err: <nil> Duration: 42.666915789s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [......X...........X.X......X..............X..X.X...X.XX..XX.] duration: 3.421142826s trace: https://jaeger.dev.storj.io/trace/238d6dc3e487bd19 FAIL (2025-12-27 13:07:07 UTC) orbiter warning: context deadline exceeded Duration: 3.421170769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X.XX..XX...X..X.X..XX.X......X..X..XXX...........XX..X...X] duration: 30.002257859s trace: https://jaeger.dev.storj.io/trace/3e65a22c50f72181 FAIL (2025-12-27 13:06:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .527615812 (525h37m20.398698847s) [upload=461059]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:06:20.527622836 (525h37m20.398705871s) [upload=461059]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:06:20.776214778 (525h37m20.647297813s) [seg_pos={0 0}, upload=461059]: Begin upload segment... Dec 27 13:06:20.776273729 (525h37m20.647356764s) [seg_pos={0 0}, upload=461059]: Done begin upload segment. Dec 27 13:06:20.776317624 (525h37m20.647400669s) [noise=true , address=45.91.116.12:4227 , piece_id=CJA7R5F6VPEPKPFMCEATKINLB3LBRAGR3QXXKPDRKRE43GF57RAA, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.776406272 (525h37m20.647489317s) [noise=true , address=145.79.227.27:4690 , piece_id=LPB7U4EOJPERXLLC22FQ2J3GBQK76Q7V66HOV2JR5DNNYOTBGNEQ, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.777058434 (525h37m20.648141479s) [noise=true , address=45.91.116.9:4262 , piece_id=MYALPSV3UOG5UIEMWZCJFCCCFEOIHYHYBTBSXGZ7H4G7QD22GSSA, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.777478313 (525h37m20.648561359s) [noise=true , address=45.91.116.5:4297 , piece_id=BQ67MTSBK5JM74BWX3UCZDYBZTB7YVSQNSKDGB5MJQFLQQELTQYA, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.778201771 (525h37m20.649284816s) [noise=true , address=77.90.31.13:4627 , piece_id=DSGALBAYUIXHHM6EWFIVFHVFJL3X3XUHHQQQLYYN3FYA3RG6HMUQ, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.778369401 (525h37m20.649452446s) [noise=true , address=145.79.227.49:4338 , piece_id=CVCGD43MZWU4JJVROXCQJEIRWLH2HH2TMLIAKQAEQDASTFOUPZIA, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.778598206 (525h37m20.649681241s) [noise=true , address=145.79.227.36:4518 , piece_id=JEAM7IQ266G6ACETDBDKJ2X7YQYZNAEYVVQBUBZZYRFM36ZHQOPQ, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.778920720 (525h37m20.650003775s) [noise=true , address=45.91.116.11:4117 , piece_id=UW6CTLZFD56SE3IHNQ77BMNJWJPQKE5EMNTJD6NDYPG2F7H4LK7A, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.778953833 (525h37m20.650036878s) [noise=true , address=77.90.31.18:4808 , piece_id=J5YT7PCGR7PVDL63DLJVRX7CFGGLZO6QUTTTSVJJ4F7EMTM3H44Q, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.779403058 (525h37m20.650486103s) [noise=true , address=145.79.227.39:4013 , piece_id=QE4Z44FQ5EKK5VI3NVCUTUHME2K5SGTNK3VBQBZ7ZWLFORSXSRTA, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.780088724 (525h37m20.651171759s) [noise=true , address=145.79.227.35:4482 , piece_id=U2CJI6F2WGOVYC4RUUZRAACQBTUS3Z4232YG2MKFMH3UKHNYHMNA, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.780879620 (525h37m20.651962655s) [noise=true , address=145.79.227.47:4267 , piece_id=PXKPNKXMEDEELKCDSWCXNTYLLFEHCYCA5BPS7JWRG5FPEWRBWWVQ, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.781616253 (525h37m20.652699298s) [noise=true , address=145.79.227.2:4113 , piece_id=TVX3MTVCZDCNLPHF35HXHPY5PCHNUBJY5CN6OI5K5ANBQBAW3T7Q, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.782303251 (525h37m20.653386286s) [noise=true , address=145.79.227.37:4566 , piece_id=NGZ3IGC2W4RUFGT4KR4NLTIPOHIRHVZFZNWC5FYPO3V7MCTRPHTA, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.782510586 (525h37m20.653593631s) [noise=true , address=145.79.227.17:4972 , piece_id=GQVFE3I2S7MHURP33UVRBXDSZ6CHWHHW7UFYHRUX7ZKM5LZJISKQ, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:20.782862105 (525h37m20.653945151s) [noise=true , address=45.91.116.4:4939 , piece_id=QNNUSDJR3VLVP5MJ23DIWBXBKFU5ODSI6APNVRQYCKSJGBSE3AGA, seg_pos={0 0}, upload=461059]: Uploading piece... Dec 27 13:06:22.487234844 (525h37m22.358317879s) [upload=461059]: Next returned nil segment Dec 27 13:06:22.487251016 (525h37m22.358334061s) [upload=461059]: Waiting for error group managing segments... Dec 27 13:06:22.774874229 (525h37m22.645957274s) [noise=true , address=145.79.227.49:4338 , piece_id=CVCGD43MZWU4JJVROXCQJEIRWLH2HH2TMLIAKQAEQDASTFOUPZIA, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:22.791298414 (525h37m22.662381459s) [noise=true , address=145.79.227.17:4972 , piece_id=GQVFE3I2S7MHURP33UVRBXDSZ6CHWHHW7UFYHRUX7ZKM5LZJISKQ, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:22.791536578 (525h37m22.662619623s) [noise=true , address=145.79.227.35:4482 , piece_id=U2CJI6F2WGOVYC4RUUZRAACQBTUS3Z4232YG2MKFMH3UKHNYHMNA, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:22.792312284 (525h37m22.66339533s) [noise=true , address=145.79.227.36:4518 , piece_id=JEAM7IQ266G6ACETDBDKJ2X7YQYZNAEYVVQBUBZZYRFM36ZHQOPQ, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:22.793712751 (525h37m22.664795786s) [noise=true , address=145.79.227.27:4690 , piece_id=LPB7U4EOJPERXLLC22FQ2J3GBQK76Q7V66HOV2JR5DNNYOTBGNEQ, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:22.803046886 (525h37m22.674129931s) [noise=true , address=145.79.227.47:4267 , piece_id=PXKPNKXMEDEELKCDSWCXNTYLLFEHCYCA5BPS7JWRG5FPEWRBWWVQ, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:22.805681602 (525h37m22.676764637s) [noise=true , address=145.79.227.37:4566 , piece_id=NGZ3IGC2W4RUFGT4KR4NLTIPOHIRHVZFZNWC5FYPO3V7MCTRPHTA, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:22.808858552 (525h37m22.679941597s) [noise=true , address=145.79.227.39:4013 , piece_id=QE4Z44FQ5EKK5VI3NVCUTUHME2K5SGTNK3VBQBZ7ZWLFORSXSRTA, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:22.811895715 (525h37m22.68297875s) [noise=true , address=145.79.227.2:4113 , piece_id=TVX3MTVCZDCNLPHF35HXHPY5PCHNUBJY5CN6OI5K5ANBQBAW3T7Q, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:22.955151839 (525h37m22.826234884s) [noise=true , address=77.90.31.18:4808 , piece_id=J5YT7PCGR7PVDL63DLJVRX7CFGGLZO6QUTTTSVJJ4F7EMTM3H44Q, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:23.030415133 (525h37m22.901498179s) [noise=true , address=45.91.116.5:4297 , piece_id=BQ67MTSBK5JM74BWX3UCZDYBZTB7YVSQNSKDGB5MJQFLQQELTQYA, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:23.087504782 (525h37m22.958587817s) [noise=true , address=77.90.31.13:4627 , piece_id=DSGALBAYUIXHHM6EWFIVFHVFJL3X3XUHHQQQLYYN3FYA3RG6HMUQ, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:23.179173705 (525h37m23.05025674s) [noise=true , address=45.91.116.12:4227 , piece_id=CJA7R5F6VPEPKPFMCEATKINLB3LBRAGR3QXXKPDRKRE43GF57RAA, seg_pos={0 0}, upload=461059]: Done uploading piece. err: <nil> Dec 27 13:06:23.179195247 (525h37m23.050278282s) [seg_pos={0 0}, upload=461059]: Segment reached optimal threshold of 13 pieces. Dec 27 13:06:23.179395368 (525h37m23.050478413s) [noise=true , address=45.91.116.4:4939 , piece_id=QNNUSDJR3VLVP5MJ23DIWBXBKFU5ODSI6APNVRQYCKSJGBSE3AGA, seg_pos={0 0}, upload=461059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:57844->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:57844->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57844->45.91.116.4:4939: use of closed network connection Dec 27 13:06:23.179403233 (525h37m23.050486268s) [noise=true , address=45.91.116.4:4939 , piece_id=QNNUSDJR3VLVP5MJ23DIWBXBKFU5ODSI6APNVRQYCKSJGBSE3AGA, seg_pos={0 0}, upload=461059]: Long tail cancellation due to optimal threshold Dec 27 13:06:23.179522379 (525h37m23.050605424s) [noise=true , address=45.91.116.9:4262 , piece_id=MYALPSV3UOG5UIEMWZCJFCCCFEOIHYHYBTBSXGZ7H4G7QD22GSSA, seg_pos={0 0}, upload=461059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:60730->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:60730->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60730->45.91.116.9:4262: use of closed network connection Dec 27 13:06:23.179530274 (525h37m23.050613319s) [noise=true , address=45.91.116.9:4262 , piece_id=MYALPSV3UOG5UIEMWZCJFCCCFEOIHYHYBTBSXGZ7H4G7QD22GSSA, seg_pos={0 0}, upload=461059]: Long tail cancellation due to optimal threshold Dec 27 13:06:23.180049202 (525h37m23.051132247s) [noise=true , address=45.91.116.11:4117 , piece_id=UW6CTLZFD56SE3IHNQ77BMNJWJPQKE5EMNTJD6NDYPG2F7H4LK7A, seg_pos={0 0}, upload=461059]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:34766->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:34766->45.91.116.11:4117: use of closed network connection Dec 27 13:06:23.180068850 (525h37m23.051151895s) [noise=true , address=45.91.116.11:4117 , piece_id=UW6CTLZFD56SE3IHNQ77BMNJWJPQKE5EMNTJD6NDYPG2F7H4LK7A, seg_pos={0 0}, upload=461059]: Long tail cancellation due to optimal threshold Dec 27 13:06:23.180135697 (525h37m23.051218742s) [seg_pos={0 0}, upload=461059]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:06:23.180283799 (525h37m23.051366834s) [upload=461059]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:06:23.180292225 (525h37m23.05137526s) [upload=461059]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:06:23.476798655 (525h37m23.3478817s) [upload=461059]: Done uploading segments... Dec 27 13:06:23.476825325 (525h37m23.34790836s) [upload=461059]: Upload finished. err: <nil> Duration: 30.002281073s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.......X...........X.X......X..............X..X.X...X.XX..XX] duration: 5.001563448s trace: https://jaeger.dev.storj.io/trace/238d6dc3e487bd19 FAIL (2025-12-27 13:06:16 UTC) orbiter warning: context deadline exceeded Duration: 5.001579709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X................................X........................] duration: 43.740880651s trace: https://jaeger.dev.storj.io/trace/67980cd8fff84f06 FAIL (2025-12-27 13:06:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sy3iQ81wRjoqXs8RhuZW677SBDn4VP6tw14hbozoWenFfgdqu, Piece ID: 5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA) context deadline exceeded; piecestore: (Node ID: 1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, Piece ID: 5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA) context deadline exceeded; piecestore: (Node ID: 124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, Piece ID: KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA) context deadline exceeded ======LOGS====== 13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461479694 (525h13m42.332562739s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461737203 (525h13m42.332820239s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462003020 (525h13m42.333086055s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462034240 (525h13m42.333117285s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462525545 (525h13m42.33360859s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462739873 (525h13m42.333822918s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462797022 (525h13m42.333880057s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463073168 (525h13m42.334156203s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463329756 (525h13m42.334412801s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463569683 (525h13m42.334652728s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463635078 (525h13m42.334718123s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463923096 (525h13m42.335006161s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.464270568 (525h13m42.335353603s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.467068917 (525h13m42.338151962s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:43.771082828 (525h13m43.642165873s) [upload=460711]: Next returned nil segment Dec 27 12:42:43.771096956 (525h13m43.642179991s) [upload=460711]: Waiting for error group managing segments... Dec 27 12:42:44.300665304 (525h13m44.171748339s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.334817064 (525h13m44.205900099s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.357754701 (525h13m44.228837736s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.370947372 (525h13m44.242030417s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.544213540 (525h13m44.415296585s) [noise=true , address=77.90.31.13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.648013103 (525h13m44.519096138s) [noise=true , address=77.90.31.11:4570 , piece_id=6BP3FAA6SZ7CMRBV3DWLDTXPE6IZ4ZD5T6CPQZ2UEKVCU4SFNF5A, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.701365715 (525h13m44.57244876s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.731017397 (525h13m44.602100442s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.184136271 (525h13m45.055219306s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.305192633 (525h13m45.176275678s) [noise=true , address=77.90.31.19:4845 , piece_id=5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.331752207 (525h13m45.202835252s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.337387216 (525h13m45.208470251s) [noise=true , address=45.91.116.13:4164 , piece_id=5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.400681550 (525h13m45.271764595s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.548984854 (525h13m45.420067899s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.549007277 (525h13m45.420090312s) [seg_pos={0 0}, upload=460711]: Segment reached optimal threshold of 14 pieces. Dec 27 12:42:45.549218289 (525h13m45.420301334s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549228118 (525h13m45.420311153s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549354449 (525h13m45.420437494s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14ZdgenD4m2bMMmaHVyK3YivrzdSnrp72JvMJsk1uecW6KKFjn): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549418640 (525h13m45.420501685s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549636385 (525h13m45.42071942s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection Dec 27 12:42:45.549643619 (525h13m45.420726664s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549904335 (525h13m45.42098738s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection Dec 27 12:42:45.549913032 (525h13m45.420996067s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549933541 (525h13m45.421016576s) [seg_pos={0 0}, upload=460711]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:42:45.550005027 (525h13m45.421088062s) [upload=460711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:42:45.550009655 (525h13m45.42109269s) [upload=460711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:42:45.852172915 (525h13m45.72325596s) [upload=460711]: Done uploading segments... Dec 27 12:42:45.852195298 (525h13m45.723278343s) [upload=460711]: Upload finished. err: <nil> Duration: 43.740900379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [........X...........X.X......X..............X..X.X...X.XX..X] duration: 5.000939409s trace: https://jaeger.dev.storj.io/trace/13d927592f00d0a7 FAIL (2025-12-27 13:05:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000963507s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX.X.XX..XX...X..X.X..XX.X......X..X..XXX...........XX..X...] duration: 5.948987047s trace: https://jaeger.dev.storj.io/trace/bc7b82d0b752e58 FAIL (2025-12-27 13:05:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 66Z7EITUPKKNXG6DWLEYNCYKW62O4CCNTUEL66UF676DIMTPGNAQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: DTVGF2DYXRTNTGDZZOF5IQE5K4TDZORYV4AABTDHMR4D65IEWAMQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: LI37CDKQFPU3QY2VG2FM7OWRUMQMJHELZTDTAAKXAE7JVXULPV3Q) context deadline exceeded ======LOGS====== ng segments... Dec 27 13:03:43.508085163 (525h34m43.379168208s) [upload=461021]: Got next segment. Inline: false Dec 27 13:03:43.508129026 (525h34m43.379212081s) [upload=461021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:03:43.508134758 (525h34m43.379217793s) [upload=461021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:03:43.755515503 (525h34m43.626598538s) [seg_pos={0 0}, upload=461021]: Begin upload segment... Dec 27 13:03:43.755560438 (525h34m43.626643473s) [seg_pos={0 0}, upload=461021]: Done begin upload segment. Dec 27 13:03:43.755595154 (525h34m43.626678189s) [noise=true , address=45.91.116.5:4299 , piece_id=DTVGF2DYXRTNTGDZZOF5IQE5K4TDZORYV4AABTDHMR4D65IEWAMQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.755696357 (525h34m43.626779402s) [noise=true , address=145.79.227.41:4085 , piece_id=DUIWCMBHKUTQAR75TPLWL7C2TWZQPF4MU7UW6AQUVIBTZ7K7NIXA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756090868 (525h34m43.627173903s) [noise=true , address=145.79.227.52:4447 , piece_id=UAO2UD75QTHKZM7MOTB2KXWQHDRV4PVUC7W72N7QUEDMU6L6CRRQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756098513 (525h34m43.627181558s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZMQ5Y7NRQNAIWO5M6MV7BMOUWHXCGC2C4NPSGGBHHBTUOROCUZQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756374579 (525h34m43.627457614s) [noise=true , address=45.91.116.7:4012 , piece_id=LI37CDKQFPU3QY2VG2FM7OWRUMQMJHELZTDTAAKXAE7JVXULPV3Q, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756430455 (525h34m43.62751349s) [noise=true , address=77.90.31.10:4519 , piece_id=ANQLWAXPRZAM3YNC7GYAYQJ2P2L6TKTRT4H7V425T5X4FBYPQ6EQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756634764 (525h34m43.627717799s) [noise=true , address=45.91.116.9:4261 , piece_id=SR6L37662MOIO6XI3MDCZMM7MZE3DYVQQWMOIUA4TFKNEW4LJ2IA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756719245 (525h34m43.62780229s) [noise=true , address=145.79.227.35:4482 , piece_id=HHZWUYFEG52B2PH7QRNZ5RM523YHCHF6FC72WGBQ27IQVAAMQVHA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756993797 (525h34m43.628076843s) [noise=true , address=45.91.116.11:4120 , piece_id=LPLOJJB6FJUDLILTLVQR3PMDWRN2KA4FQ37CZ6NWOVKZQCIYPC3A, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757064191 (525h34m43.628147227s) [noise=true , address=145.79.227.48:4302 , piece_id=UEXBPBPZRDU62WSUQ4YXQBVYXSDTTWS4MYPUL42PUPTCABFATJUA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757305952 (525h34m43.628388987s) [noise=true , address=145.79.227.36:4517 , piece_id=2W4DJXMAPDIWJVDAW3J2RCFVPU7MXXKK6QVMJRUBAJKDDPCQBLXA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757353934 (525h34m43.628436979s) [noise=true , address=145.79.227.25:4617 , piece_id=EZHMCPA4OL3G54BBARBTZVGAEQY5JAG5HJWSEGD2PXZSHJFBVGRA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757563783 (525h34m43.628646829s) [noise=true , address=145.79.227.50:4373 , piece_id=DDQR5GOC6E2227SPJDN7IOI5XUALTHL2PY73TK2RGT4BY6QRGO4Q, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757655117 (525h34m43.628738152s) [noise=true , address=45.91.116.10:4082 , piece_id=66Z7EITUPKKNXG6DWLEYNCYKW62O4CCNTUEL66UF676DIMTPGNAQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757855238 (525h34m43.628938273s) [noise=true , address=145.79.227.39:4014 , piece_id=QXPAP3YB5ASRU2I66WWX5MLALOFPXIP3QEV7KEVNY2WC7Z6ALZCQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757995475 (525h34m43.62907852s) [noise=true , address=145.79.227.49:4339 , piece_id=MEBKJDKVADF6E2DU422PPUE4NAG4XUDCGT7DCQO2NQLJTR76JC2Q, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:45.476103088 (525h34m45.347186133s) [upload=461021]: Next returned nil segment Dec 27 13:03:45.476120642 (525h34m45.347203687s) [upload=461021]: Waiting for error group managing segments... Dec 27 13:03:45.760277926 (525h34m45.631360971s) [noise=true , address=145.79.227.25:4617 , piece_id=EZHMCPA4OL3G54BBARBTZVGAEQY5JAG5HJWSEGD2PXZSHJFBVGRA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.771720365 (525h34m45.6428034s) [noise=true , address=145.79.227.52:4447 , piece_id=UAO2UD75QTHKZM7MOTB2KXWQHDRV4PVUC7W72N7QUEDMU6L6CRRQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.777751239 (525h34m45.648834284s) [noise=true , address=145.79.227.39:4014 , piece_id=QXPAP3YB5ASRU2I66WWX5MLALOFPXIP3QEV7KEVNY2WC7Z6ALZCQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.777787488 (525h34m45.648870523s) [noise=true , address=145.79.227.41:4085 , piece_id=DUIWCMBHKUTQAR75TPLWL7C2TWZQPF4MU7UW6AQUVIBTZ7K7NIXA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.781443319 (525h34m45.652526354s) [noise=true , address=145.79.227.49:4339 , piece_id=MEBKJDKVADF6E2DU422PPUE4NAG4XUDCGT7DCQO2NQLJTR76JC2Q, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.781702642 (525h34m45.652785677s) [noise=true , address=145.79.227.48:4302 , piece_id=UEXBPBPZRDU62WSUQ4YXQBVYXSDTTWS4MYPUL42PUPTCABFATJUA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.783234930 (525h34m45.654317975s) [noise=true , address=145.79.227.50:4373 , piece_id=DDQR5GOC6E2227SPJDN7IOI5XUALTHL2PY73TK2RGT4BY6QRGO4Q, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.788660451 (525h34m45.659743486s) [noise=true , address=145.79.227.36:4517 , piece_id=2W4DJXMAPDIWJVDAW3J2RCFVPU7MXXKK6QVMJRUBAJKDDPCQBLXA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.792474123 (525h34m45.663557158s) [noise=true , address=145.79.227.35:4482 , piece_id=HHZWUYFEG52B2PH7QRNZ5RM523YHCHF6FC72WGBQ27IQVAAMQVHA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.996308238 (525h34m45.867391274s) [noise=true , address=45.91.116.7:4012 , piece_id=LI37CDKQFPU3QY2VG2FM7OWRUMQMJHELZTDTAAKXAE7JVXULPV3Q, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.996693712 (525h34m45.867776747s) [noise=true , address=45.91.116.5:4299 , piece_id=DTVGF2DYXRTNTGDZZOF5IQE5K4TDZORYV4AABTDHMR4D65IEWAMQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.997658259 (525h34m45.868741294s) [noise=true , address=45.91.116.10:4082 , piece_id=66Z7EITUPKKNXG6DWLEYNCYKW62O4CCNTUEL66UF676DIMTPGNAQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:46.021556505 (525h34m45.89263955s) [noise=true , address=45.91.116.11:4120 , piece_id=LPLOJJB6FJUDLILTLVQR3PMDWRN2KA4FQ37CZ6NWOVKZQCIYPC3A, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:46.021578117 (525h34m45.892661162s) [seg_pos={0 0}, upload=461021]: Segment reached optimal threshold of 13 pieces. Dec 27 13:03:46.021758440 (525h34m45.892841475s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZMQ5Y7NRQNAIWO5M6MV7BMOUWHXCGC2C4NPSGGBHHBTUOROCUZQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 13:03:46.021765012 (525h34m45.892848047s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZMQ5Y7NRQNAIWO5M6MV7BMOUWHXCGC2C4NPSGGBHHBTUOROCUZQ, seg_pos={0 0}, upload=461021]: Long tail cancellation due to optimal threshold Dec 27 13:03:46.021990362 (525h34m45.893073397s) [noise=true , address=45.91.116.9:4261 , piece_id=SR6L37662MOIO6XI3MDCZMM7MZE3DYVQQWMOIUA4TFKNEW4LJ2IA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:37312->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:37312->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->45.91.116.9:4261: use of closed network connection Dec 27 13:03:46.021997916 (525h34m45.893080961s) [noise=true , address=45.91.116.9:4261 , piece_id=SR6L37662MOIO6XI3MDCZMM7MZE3DYVQQWMOIUA4TFKNEW4LJ2IA, seg_pos={0 0}, upload=461021]: Long tail cancellation due to optimal threshold Dec 27 13:03:46.022004799 (525h34m45.893087834s) [noise=true , address=77.90.31.10:4519 , piece_id=ANQLWAXPRZAM3YNC7GYAYQJ2P2L6TKTRT4H7V425T5X4FBYPQ6EQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:57404->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:57404->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57404->77.90.31.10:4519: use of closed network connection Dec 27 13:03:46.022011151 (525h34m45.893094196s) [noise=true , address=77.90.31.10:4519 , piece_id=ANQLWAXPRZAM3YNC7GYAYQJ2P2L6TKTRT4H7V425T5X4FBYPQ6EQ, seg_pos={0 0}, upload=461021]: Long tail cancellation due to optimal threshold Dec 27 13:03:46.022168371 (525h34m45.893251406s) [seg_pos={0 0}, upload=461021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:03:46.022237903 (525h34m45.893320938s) [upload=461021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:03:46.022241891 (525h34m45.893324936s) [upload=461021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:03:46.317533958 (525h34m46.188616993s) [upload=461021]: Done uploading segments... Dec 27 13:03:46.317554166 (525h34m46.188637201s) [upload=461021]: Upload finished. err: <nil> Duration: 5.949011564s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.........X...........X.X......X..............X..X.X...X.XX..] duration: 3.789364553s trace: https://jaeger.dev.storj.io/trace/27eb7c025fd2e151 FAIL (2025-12-27 13:05:30 UTC) orbiter warning: context deadline exceeded Duration: 3.789471536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X................................X.......................] duration: 47.887017092s trace: https://jaeger.dev.storj.io/trace/67980cd8fff84f06 FAIL (2025-12-27 13:05:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sy3iQ81wRjoqXs8RhuZW677SBDn4VP6tw14hbozoWenFfgdqu, Piece ID: 5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA) context deadline exceeded; piecestore: (Node ID: 1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, Piece ID: 5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA) context deadline exceeded; piecestore: (Node ID: 124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, Piece ID: KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA) context deadline exceeded ======LOGS====== 13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461479694 (525h13m42.332562739s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461737203 (525h13m42.332820239s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462003020 (525h13m42.333086055s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462034240 (525h13m42.333117285s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462525545 (525h13m42.33360859s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462739873 (525h13m42.333822918s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462797022 (525h13m42.333880057s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463073168 (525h13m42.334156203s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463329756 (525h13m42.334412801s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463569683 (525h13m42.334652728s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463635078 (525h13m42.334718123s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463923096 (525h13m42.335006161s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.464270568 (525h13m42.335353603s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.467068917 (525h13m42.338151962s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:43.771082828 (525h13m43.642165873s) [upload=460711]: Next returned nil segment Dec 27 12:42:43.771096956 (525h13m43.642179991s) [upload=460711]: Waiting for error group managing segments... Dec 27 12:42:44.300665304 (525h13m44.171748339s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.334817064 (525h13m44.205900099s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.357754701 (525h13m44.228837736s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.370947372 (525h13m44.242030417s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.544213540 (525h13m44.415296585s) [noise=true , address=77.90.31.13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.648013103 (525h13m44.519096138s) [noise=true , address=77.90.31.11:4570 , piece_id=6BP3FAA6SZ7CMRBV3DWLDTXPE6IZ4ZD5T6CPQZ2UEKVCU4SFNF5A, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.701365715 (525h13m44.57244876s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.731017397 (525h13m44.602100442s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.184136271 (525h13m45.055219306s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.305192633 (525h13m45.176275678s) [noise=true , address=77.90.31.19:4845 , piece_id=5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.331752207 (525h13m45.202835252s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.337387216 (525h13m45.208470251s) [noise=true , address=45.91.116.13:4164 , piece_id=5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.400681550 (525h13m45.271764595s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.548984854 (525h13m45.420067899s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.549007277 (525h13m45.420090312s) [seg_pos={0 0}, upload=460711]: Segment reached optimal threshold of 14 pieces. Dec 27 12:42:45.549218289 (525h13m45.420301334s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549228118 (525h13m45.420311153s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549354449 (525h13m45.420437494s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14ZdgenD4m2bMMmaHVyK3YivrzdSnrp72JvMJsk1uecW6KKFjn): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549418640 (525h13m45.420501685s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549636385 (525h13m45.42071942s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection Dec 27 12:42:45.549643619 (525h13m45.420726664s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549904335 (525h13m45.42098738s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection Dec 27 12:42:45.549913032 (525h13m45.420996067s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549933541 (525h13m45.421016576s) [seg_pos={0 0}, upload=460711]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:42:45.550005027 (525h13m45.421088062s) [upload=460711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:42:45.550009655 (525h13m45.42109269s) [upload=460711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:42:45.852172915 (525h13m45.72325596s) [upload=460711]: Done uploading segments... Dec 27 12:42:45.852195298 (525h13m45.723278343s) [upload=460711]: Upload finished. err: <nil> Duration: 47.887045337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XX.X.XX..XX...X..X.X..XX.X......X..X..XXX...........XX..X..] duration: 6.031543487s trace: https://jaeger.dev.storj.io/trace/bc7b82d0b752e58 FAIL (2025-12-27 13:04:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 66Z7EITUPKKNXG6DWLEYNCYKW62O4CCNTUEL66UF676DIMTPGNAQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: DTVGF2DYXRTNTGDZZOF5IQE5K4TDZORYV4AABTDHMR4D65IEWAMQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: LI37CDKQFPU3QY2VG2FM7OWRUMQMJHELZTDTAAKXAE7JVXULPV3Q) context deadline exceeded ======LOGS====== ng segments... Dec 27 13:03:43.508085163 (525h34m43.379168208s) [upload=461021]: Got next segment. Inline: false Dec 27 13:03:43.508129026 (525h34m43.379212081s) [upload=461021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:03:43.508134758 (525h34m43.379217793s) [upload=461021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:03:43.755515503 (525h34m43.626598538s) [seg_pos={0 0}, upload=461021]: Begin upload segment... Dec 27 13:03:43.755560438 (525h34m43.626643473s) [seg_pos={0 0}, upload=461021]: Done begin upload segment. Dec 27 13:03:43.755595154 (525h34m43.626678189s) [noise=true , address=45.91.116.5:4299 , piece_id=DTVGF2DYXRTNTGDZZOF5IQE5K4TDZORYV4AABTDHMR4D65IEWAMQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.755696357 (525h34m43.626779402s) [noise=true , address=145.79.227.41:4085 , piece_id=DUIWCMBHKUTQAR75TPLWL7C2TWZQPF4MU7UW6AQUVIBTZ7K7NIXA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756090868 (525h34m43.627173903s) [noise=true , address=145.79.227.52:4447 , piece_id=UAO2UD75QTHKZM7MOTB2KXWQHDRV4PVUC7W72N7QUEDMU6L6CRRQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756098513 (525h34m43.627181558s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZMQ5Y7NRQNAIWO5M6MV7BMOUWHXCGC2C4NPSGGBHHBTUOROCUZQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756374579 (525h34m43.627457614s) [noise=true , address=45.91.116.7:4012 , piece_id=LI37CDKQFPU3QY2VG2FM7OWRUMQMJHELZTDTAAKXAE7JVXULPV3Q, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756430455 (525h34m43.62751349s) [noise=true , address=77.90.31.10:4519 , piece_id=ANQLWAXPRZAM3YNC7GYAYQJ2P2L6TKTRT4H7V425T5X4FBYPQ6EQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756634764 (525h34m43.627717799s) [noise=true , address=45.91.116.9:4261 , piece_id=SR6L37662MOIO6XI3MDCZMM7MZE3DYVQQWMOIUA4TFKNEW4LJ2IA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756719245 (525h34m43.62780229s) [noise=true , address=145.79.227.35:4482 , piece_id=HHZWUYFEG52B2PH7QRNZ5RM523YHCHF6FC72WGBQ27IQVAAMQVHA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756993797 (525h34m43.628076843s) [noise=true , address=45.91.116.11:4120 , piece_id=LPLOJJB6FJUDLILTLVQR3PMDWRN2KA4FQ37CZ6NWOVKZQCIYPC3A, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757064191 (525h34m43.628147227s) [noise=true , address=145.79.227.48:4302 , piece_id=UEXBPBPZRDU62WSUQ4YXQBVYXSDTTWS4MYPUL42PUPTCABFATJUA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757305952 (525h34m43.628388987s) [noise=true , address=145.79.227.36:4517 , piece_id=2W4DJXMAPDIWJVDAW3J2RCFVPU7MXXKK6QVMJRUBAJKDDPCQBLXA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757353934 (525h34m43.628436979s) [noise=true , address=145.79.227.25:4617 , piece_id=EZHMCPA4OL3G54BBARBTZVGAEQY5JAG5HJWSEGD2PXZSHJFBVGRA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757563783 (525h34m43.628646829s) [noise=true , address=145.79.227.50:4373 , piece_id=DDQR5GOC6E2227SPJDN7IOI5XUALTHL2PY73TK2RGT4BY6QRGO4Q, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757655117 (525h34m43.628738152s) [noise=true , address=45.91.116.10:4082 , piece_id=66Z7EITUPKKNXG6DWLEYNCYKW62O4CCNTUEL66UF676DIMTPGNAQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757855238 (525h34m43.628938273s) [noise=true , address=145.79.227.39:4014 , piece_id=QXPAP3YB5ASRU2I66WWX5MLALOFPXIP3QEV7KEVNY2WC7Z6ALZCQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757995475 (525h34m43.62907852s) [noise=true , address=145.79.227.49:4339 , piece_id=MEBKJDKVADF6E2DU422PPUE4NAG4XUDCGT7DCQO2NQLJTR76JC2Q, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:45.476103088 (525h34m45.347186133s) [upload=461021]: Next returned nil segment Dec 27 13:03:45.476120642 (525h34m45.347203687s) [upload=461021]: Waiting for error group managing segments... Dec 27 13:03:45.760277926 (525h34m45.631360971s) [noise=true , address=145.79.227.25:4617 , piece_id=EZHMCPA4OL3G54BBARBTZVGAEQY5JAG5HJWSEGD2PXZSHJFBVGRA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.771720365 (525h34m45.6428034s) [noise=true , address=145.79.227.52:4447 , piece_id=UAO2UD75QTHKZM7MOTB2KXWQHDRV4PVUC7W72N7QUEDMU6L6CRRQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.777751239 (525h34m45.648834284s) [noise=true , address=145.79.227.39:4014 , piece_id=QXPAP3YB5ASRU2I66WWX5MLALOFPXIP3QEV7KEVNY2WC7Z6ALZCQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.777787488 (525h34m45.648870523s) [noise=true , address=145.79.227.41:4085 , piece_id=DUIWCMBHKUTQAR75TPLWL7C2TWZQPF4MU7UW6AQUVIBTZ7K7NIXA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.781443319 (525h34m45.652526354s) [noise=true , address=145.79.227.49:4339 , piece_id=MEBKJDKVADF6E2DU422PPUE4NAG4XUDCGT7DCQO2NQLJTR76JC2Q, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.781702642 (525h34m45.652785677s) [noise=true , address=145.79.227.48:4302 , piece_id=UEXBPBPZRDU62WSUQ4YXQBVYXSDTTWS4MYPUL42PUPTCABFATJUA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.783234930 (525h34m45.654317975s) [noise=true , address=145.79.227.50:4373 , piece_id=DDQR5GOC6E2227SPJDN7IOI5XUALTHL2PY73TK2RGT4BY6QRGO4Q, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.788660451 (525h34m45.659743486s) [noise=true , address=145.79.227.36:4517 , piece_id=2W4DJXMAPDIWJVDAW3J2RCFVPU7MXXKK6QVMJRUBAJKDDPCQBLXA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.792474123 (525h34m45.663557158s) [noise=true , address=145.79.227.35:4482 , piece_id=HHZWUYFEG52B2PH7QRNZ5RM523YHCHF6FC72WGBQ27IQVAAMQVHA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.996308238 (525h34m45.867391274s) [noise=true , address=45.91.116.7:4012 , piece_id=LI37CDKQFPU3QY2VG2FM7OWRUMQMJHELZTDTAAKXAE7JVXULPV3Q, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.996693712 (525h34m45.867776747s) [noise=true , address=45.91.116.5:4299 , piece_id=DTVGF2DYXRTNTGDZZOF5IQE5K4TDZORYV4AABTDHMR4D65IEWAMQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.997658259 (525h34m45.868741294s) [noise=true , address=45.91.116.10:4082 , piece_id=66Z7EITUPKKNXG6DWLEYNCYKW62O4CCNTUEL66UF676DIMTPGNAQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:46.021556505 (525h34m45.89263955s) [noise=true , address=45.91.116.11:4120 , piece_id=LPLOJJB6FJUDLILTLVQR3PMDWRN2KA4FQ37CZ6NWOVKZQCIYPC3A, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:46.021578117 (525h34m45.892661162s) [seg_pos={0 0}, upload=461021]: Segment reached optimal threshold of 13 pieces. Dec 27 13:03:46.021758440 (525h34m45.892841475s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZMQ5Y7NRQNAIWO5M6MV7BMOUWHXCGC2C4NPSGGBHHBTUOROCUZQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 13:03:46.021765012 (525h34m45.892848047s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZMQ5Y7NRQNAIWO5M6MV7BMOUWHXCGC2C4NPSGGBHHBTUOROCUZQ, seg_pos={0 0}, upload=461021]: Long tail cancellation due to optimal threshold Dec 27 13:03:46.021990362 (525h34m45.893073397s) [noise=true , address=45.91.116.9:4261 , piece_id=SR6L37662MOIO6XI3MDCZMM7MZE3DYVQQWMOIUA4TFKNEW4LJ2IA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:37312->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:37312->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->45.91.116.9:4261: use of closed network connection Dec 27 13:03:46.021997916 (525h34m45.893080961s) [noise=true , address=45.91.116.9:4261 , piece_id=SR6L37662MOIO6XI3MDCZMM7MZE3DYVQQWMOIUA4TFKNEW4LJ2IA, seg_pos={0 0}, upload=461021]: Long tail cancellation due to optimal threshold Dec 27 13:03:46.022004799 (525h34m45.893087834s) [noise=true , address=77.90.31.10:4519 , piece_id=ANQLWAXPRZAM3YNC7GYAYQJ2P2L6TKTRT4H7V425T5X4FBYPQ6EQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:57404->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:57404->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57404->77.90.31.10:4519: use of closed network connection Dec 27 13:03:46.022011151 (525h34m45.893094196s) [noise=true , address=77.90.31.10:4519 , piece_id=ANQLWAXPRZAM3YNC7GYAYQJ2P2L6TKTRT4H7V425T5X4FBYPQ6EQ, seg_pos={0 0}, upload=461021]: Long tail cancellation due to optimal threshold Dec 27 13:03:46.022168371 (525h34m45.893251406s) [seg_pos={0 0}, upload=461021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:03:46.022237903 (525h34m45.893320938s) [upload=461021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:03:46.022241891 (525h34m45.893324936s) [upload=461021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:03:46.317533958 (525h34m46.188616993s) [upload=461021]: Done uploading segments... Dec 27 13:03:46.317554166 (525h34m46.188637201s) [upload=461021]: Upload finished. err: <nil> Duration: 6.031574486s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..........X...........X.X......X..............X..X.X...X.XX.] duration: 3.907092177s trace: https://jaeger.dev.storj.io/trace/27eb7c025fd2e151 FAIL (2025-12-27 13:04:46 UTC) orbiter warning: context deadline exceeded Duration: 3.907121683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XX.X.XX..XX...X..X.X..XX.X......X..X..XXX...........XX..X.] duration: 6.641707645s trace: https://jaeger.dev.storj.io/trace/bc7b82d0b752e58 FAIL (2025-12-27 13:04:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 66Z7EITUPKKNXG6DWLEYNCYKW62O4CCNTUEL66UF676DIMTPGNAQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: DTVGF2DYXRTNTGDZZOF5IQE5K4TDZORYV4AABTDHMR4D65IEWAMQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: LI37CDKQFPU3QY2VG2FM7OWRUMQMJHELZTDTAAKXAE7JVXULPV3Q) context deadline exceeded ======LOGS====== ng segments... Dec 27 13:03:43.508085163 (525h34m43.379168208s) [upload=461021]: Got next segment. Inline: false Dec 27 13:03:43.508129026 (525h34m43.379212081s) [upload=461021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:03:43.508134758 (525h34m43.379217793s) [upload=461021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:03:43.755515503 (525h34m43.626598538s) [seg_pos={0 0}, upload=461021]: Begin upload segment... Dec 27 13:03:43.755560438 (525h34m43.626643473s) [seg_pos={0 0}, upload=461021]: Done begin upload segment. Dec 27 13:03:43.755595154 (525h34m43.626678189s) [noise=true , address=45.91.116.5:4299 , piece_id=DTVGF2DYXRTNTGDZZOF5IQE5K4TDZORYV4AABTDHMR4D65IEWAMQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.755696357 (525h34m43.626779402s) [noise=true , address=145.79.227.41:4085 , piece_id=DUIWCMBHKUTQAR75TPLWL7C2TWZQPF4MU7UW6AQUVIBTZ7K7NIXA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756090868 (525h34m43.627173903s) [noise=true , address=145.79.227.52:4447 , piece_id=UAO2UD75QTHKZM7MOTB2KXWQHDRV4PVUC7W72N7QUEDMU6L6CRRQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756098513 (525h34m43.627181558s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZMQ5Y7NRQNAIWO5M6MV7BMOUWHXCGC2C4NPSGGBHHBTUOROCUZQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756374579 (525h34m43.627457614s) [noise=true , address=45.91.116.7:4012 , piece_id=LI37CDKQFPU3QY2VG2FM7OWRUMQMJHELZTDTAAKXAE7JVXULPV3Q, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756430455 (525h34m43.62751349s) [noise=true , address=77.90.31.10:4519 , piece_id=ANQLWAXPRZAM3YNC7GYAYQJ2P2L6TKTRT4H7V425T5X4FBYPQ6EQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756634764 (525h34m43.627717799s) [noise=true , address=45.91.116.9:4261 , piece_id=SR6L37662MOIO6XI3MDCZMM7MZE3DYVQQWMOIUA4TFKNEW4LJ2IA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756719245 (525h34m43.62780229s) [noise=true , address=145.79.227.35:4482 , piece_id=HHZWUYFEG52B2PH7QRNZ5RM523YHCHF6FC72WGBQ27IQVAAMQVHA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756993797 (525h34m43.628076843s) [noise=true , address=45.91.116.11:4120 , piece_id=LPLOJJB6FJUDLILTLVQR3PMDWRN2KA4FQ37CZ6NWOVKZQCIYPC3A, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757064191 (525h34m43.628147227s) [noise=true , address=145.79.227.48:4302 , piece_id=UEXBPBPZRDU62WSUQ4YXQBVYXSDTTWS4MYPUL42PUPTCABFATJUA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757305952 (525h34m43.628388987s) [noise=true , address=145.79.227.36:4517 , piece_id=2W4DJXMAPDIWJVDAW3J2RCFVPU7MXXKK6QVMJRUBAJKDDPCQBLXA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757353934 (525h34m43.628436979s) [noise=true , address=145.79.227.25:4617 , piece_id=EZHMCPA4OL3G54BBARBTZVGAEQY5JAG5HJWSEGD2PXZSHJFBVGRA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757563783 (525h34m43.628646829s) [noise=true , address=145.79.227.50:4373 , piece_id=DDQR5GOC6E2227SPJDN7IOI5XUALTHL2PY73TK2RGT4BY6QRGO4Q, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757655117 (525h34m43.628738152s) [noise=true , address=45.91.116.10:4082 , piece_id=66Z7EITUPKKNXG6DWLEYNCYKW62O4CCNTUEL66UF676DIMTPGNAQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757855238 (525h34m43.628938273s) [noise=true , address=145.79.227.39:4014 , piece_id=QXPAP3YB5ASRU2I66WWX5MLALOFPXIP3QEV7KEVNY2WC7Z6ALZCQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757995475 (525h34m43.62907852s) [noise=true , address=145.79.227.49:4339 , piece_id=MEBKJDKVADF6E2DU422PPUE4NAG4XUDCGT7DCQO2NQLJTR76JC2Q, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:45.476103088 (525h34m45.347186133s) [upload=461021]: Next returned nil segment Dec 27 13:03:45.476120642 (525h34m45.347203687s) [upload=461021]: Waiting for error group managing segments... Dec 27 13:03:45.760277926 (525h34m45.631360971s) [noise=true , address=145.79.227.25:4617 , piece_id=EZHMCPA4OL3G54BBARBTZVGAEQY5JAG5HJWSEGD2PXZSHJFBVGRA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.771720365 (525h34m45.6428034s) [noise=true , address=145.79.227.52:4447 , piece_id=UAO2UD75QTHKZM7MOTB2KXWQHDRV4PVUC7W72N7QUEDMU6L6CRRQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.777751239 (525h34m45.648834284s) [noise=true , address=145.79.227.39:4014 , piece_id=QXPAP3YB5ASRU2I66WWX5MLALOFPXIP3QEV7KEVNY2WC7Z6ALZCQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.777787488 (525h34m45.648870523s) [noise=true , address=145.79.227.41:4085 , piece_id=DUIWCMBHKUTQAR75TPLWL7C2TWZQPF4MU7UW6AQUVIBTZ7K7NIXA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.781443319 (525h34m45.652526354s) [noise=true , address=145.79.227.49:4339 , piece_id=MEBKJDKVADF6E2DU422PPUE4NAG4XUDCGT7DCQO2NQLJTR76JC2Q, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.781702642 (525h34m45.652785677s) [noise=true , address=145.79.227.48:4302 , piece_id=UEXBPBPZRDU62WSUQ4YXQBVYXSDTTWS4MYPUL42PUPTCABFATJUA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.783234930 (525h34m45.654317975s) [noise=true , address=145.79.227.50:4373 , piece_id=DDQR5GOC6E2227SPJDN7IOI5XUALTHL2PY73TK2RGT4BY6QRGO4Q, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.788660451 (525h34m45.659743486s) [noise=true , address=145.79.227.36:4517 , piece_id=2W4DJXMAPDIWJVDAW3J2RCFVPU7MXXKK6QVMJRUBAJKDDPCQBLXA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.792474123 (525h34m45.663557158s) [noise=true , address=145.79.227.35:4482 , piece_id=HHZWUYFEG52B2PH7QRNZ5RM523YHCHF6FC72WGBQ27IQVAAMQVHA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.996308238 (525h34m45.867391274s) [noise=true , address=45.91.116.7:4012 , piece_id=LI37CDKQFPU3QY2VG2FM7OWRUMQMJHELZTDTAAKXAE7JVXULPV3Q, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.996693712 (525h34m45.867776747s) [noise=true , address=45.91.116.5:4299 , piece_id=DTVGF2DYXRTNTGDZZOF5IQE5K4TDZORYV4AABTDHMR4D65IEWAMQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.997658259 (525h34m45.868741294s) [noise=true , address=45.91.116.10:4082 , piece_id=66Z7EITUPKKNXG6DWLEYNCYKW62O4CCNTUEL66UF676DIMTPGNAQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:46.021556505 (525h34m45.89263955s) [noise=true , address=45.91.116.11:4120 , piece_id=LPLOJJB6FJUDLILTLVQR3PMDWRN2KA4FQ37CZ6NWOVKZQCIYPC3A, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:46.021578117 (525h34m45.892661162s) [seg_pos={0 0}, upload=461021]: Segment reached optimal threshold of 13 pieces. Dec 27 13:03:46.021758440 (525h34m45.892841475s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZMQ5Y7NRQNAIWO5M6MV7BMOUWHXCGC2C4NPSGGBHHBTUOROCUZQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 13:03:46.021765012 (525h34m45.892848047s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZMQ5Y7NRQNAIWO5M6MV7BMOUWHXCGC2C4NPSGGBHHBTUOROCUZQ, seg_pos={0 0}, upload=461021]: Long tail cancellation due to optimal threshold Dec 27 13:03:46.021990362 (525h34m45.893073397s) [noise=true , address=45.91.116.9:4261 , piece_id=SR6L37662MOIO6XI3MDCZMM7MZE3DYVQQWMOIUA4TFKNEW4LJ2IA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:37312->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:37312->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->45.91.116.9:4261: use of closed network connection Dec 27 13:03:46.021997916 (525h34m45.893080961s) [noise=true , address=45.91.116.9:4261 , piece_id=SR6L37662MOIO6XI3MDCZMM7MZE3DYVQQWMOIUA4TFKNEW4LJ2IA, seg_pos={0 0}, upload=461021]: Long tail cancellation due to optimal threshold Dec 27 13:03:46.022004799 (525h34m45.893087834s) [noise=true , address=77.90.31.10:4519 , piece_id=ANQLWAXPRZAM3YNC7GYAYQJ2P2L6TKTRT4H7V425T5X4FBYPQ6EQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:57404->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:57404->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57404->77.90.31.10:4519: use of closed network connection Dec 27 13:03:46.022011151 (525h34m45.893094196s) [noise=true , address=77.90.31.10:4519 , piece_id=ANQLWAXPRZAM3YNC7GYAYQJ2P2L6TKTRT4H7V425T5X4FBYPQ6EQ, seg_pos={0 0}, upload=461021]: Long tail cancellation due to optimal threshold Dec 27 13:03:46.022168371 (525h34m45.893251406s) [seg_pos={0 0}, upload=461021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:03:46.022237903 (525h34m45.893320938s) [upload=461021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:03:46.022241891 (525h34m45.893324936s) [upload=461021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:03:46.317533958 (525h34m46.188616993s) [upload=461021]: Done uploading segments... Dec 27 13:03:46.317554166 (525h34m46.188637201s) [upload=461021]: Upload finished. err: <nil> Duration: 6.641725388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X................................X......................] duration: 42.562566281s trace: https://jaeger.dev.storj.io/trace/67980cd8fff84f06 FAIL (2025-12-27 13:04:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sy3iQ81wRjoqXs8RhuZW677SBDn4VP6tw14hbozoWenFfgdqu, Piece ID: 5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA) context deadline exceeded; piecestore: (Node ID: 1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, Piece ID: 5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA) context deadline exceeded; piecestore: (Node ID: 124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, Piece ID: KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA) context deadline exceeded ======LOGS====== 13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461479694 (525h13m42.332562739s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461737203 (525h13m42.332820239s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462003020 (525h13m42.333086055s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462034240 (525h13m42.333117285s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462525545 (525h13m42.33360859s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462739873 (525h13m42.333822918s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462797022 (525h13m42.333880057s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463073168 (525h13m42.334156203s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463329756 (525h13m42.334412801s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463569683 (525h13m42.334652728s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463635078 (525h13m42.334718123s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463923096 (525h13m42.335006161s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.464270568 (525h13m42.335353603s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.467068917 (525h13m42.338151962s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:43.771082828 (525h13m43.642165873s) [upload=460711]: Next returned nil segment Dec 27 12:42:43.771096956 (525h13m43.642179991s) [upload=460711]: Waiting for error group managing segments... Dec 27 12:42:44.300665304 (525h13m44.171748339s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.334817064 (525h13m44.205900099s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.357754701 (525h13m44.228837736s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.370947372 (525h13m44.242030417s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.544213540 (525h13m44.415296585s) [noise=true , address=77.90.31.13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.648013103 (525h13m44.519096138s) [noise=true , address=77.90.31.11:4570 , piece_id=6BP3FAA6SZ7CMRBV3DWLDTXPE6IZ4ZD5T6CPQZ2UEKVCU4SFNF5A, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.701365715 (525h13m44.57244876s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.731017397 (525h13m44.602100442s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.184136271 (525h13m45.055219306s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.305192633 (525h13m45.176275678s) [noise=true , address=77.90.31.19:4845 , piece_id=5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.331752207 (525h13m45.202835252s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.337387216 (525h13m45.208470251s) [noise=true , address=45.91.116.13:4164 , piece_id=5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.400681550 (525h13m45.271764595s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.548984854 (525h13m45.420067899s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.549007277 (525h13m45.420090312s) [seg_pos={0 0}, upload=460711]: Segment reached optimal threshold of 14 pieces. Dec 27 12:42:45.549218289 (525h13m45.420301334s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549228118 (525h13m45.420311153s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549354449 (525h13m45.420437494s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14ZdgenD4m2bMMmaHVyK3YivrzdSnrp72JvMJsk1uecW6KKFjn): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549418640 (525h13m45.420501685s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549636385 (525h13m45.42071942s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection Dec 27 12:42:45.549643619 (525h13m45.420726664s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549904335 (525h13m45.42098738s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection Dec 27 12:42:45.549913032 (525h13m45.420996067s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549933541 (525h13m45.421016576s) [seg_pos={0 0}, upload=460711]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:42:45.550005027 (525h13m45.421088062s) [upload=460711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:42:45.550009655 (525h13m45.42109269s) [upload=460711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:42:45.852172915 (525h13m45.72325596s) [upload=460711]: Done uploading segments... Dec 27 12:42:45.852195298 (525h13m45.723278343s) [upload=460711]: Upload finished. err: <nil> Duration: 42.562593863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...XX.X.XX..XX...X..X.X..XX.X......X..X..XXX...........XX..X] duration: 30.004433592s trace: https://jaeger.dev.storj.io/trace/bc7b82d0b752e58 FAIL (2025-12-27 13:04:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: 66Z7EITUPKKNXG6DWLEYNCYKW62O4CCNTUEL66UF676DIMTPGNAQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: DTVGF2DYXRTNTGDZZOF5IQE5K4TDZORYV4AABTDHMR4D65IEWAMQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: LI37CDKQFPU3QY2VG2FM7OWRUMQMJHELZTDTAAKXAE7JVXULPV3Q) context deadline exceeded ======LOGS====== ng segments... Dec 27 13:03:43.508085163 (525h34m43.379168208s) [upload=461021]: Got next segment. Inline: false Dec 27 13:03:43.508129026 (525h34m43.379212081s) [upload=461021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 13:03:43.508134758 (525h34m43.379217793s) [upload=461021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 13:03:43.755515503 (525h34m43.626598538s) [seg_pos={0 0}, upload=461021]: Begin upload segment... Dec 27 13:03:43.755560438 (525h34m43.626643473s) [seg_pos={0 0}, upload=461021]: Done begin upload segment. Dec 27 13:03:43.755595154 (525h34m43.626678189s) [noise=true , address=45.91.116.5:4299 , piece_id=DTVGF2DYXRTNTGDZZOF5IQE5K4TDZORYV4AABTDHMR4D65IEWAMQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.755696357 (525h34m43.626779402s) [noise=true , address=145.79.227.41:4085 , piece_id=DUIWCMBHKUTQAR75TPLWL7C2TWZQPF4MU7UW6AQUVIBTZ7K7NIXA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756090868 (525h34m43.627173903s) [noise=true , address=145.79.227.52:4447 , piece_id=UAO2UD75QTHKZM7MOTB2KXWQHDRV4PVUC7W72N7QUEDMU6L6CRRQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756098513 (525h34m43.627181558s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZMQ5Y7NRQNAIWO5M6MV7BMOUWHXCGC2C4NPSGGBHHBTUOROCUZQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756374579 (525h34m43.627457614s) [noise=true , address=45.91.116.7:4012 , piece_id=LI37CDKQFPU3QY2VG2FM7OWRUMQMJHELZTDTAAKXAE7JVXULPV3Q, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756430455 (525h34m43.62751349s) [noise=true , address=77.90.31.10:4519 , piece_id=ANQLWAXPRZAM3YNC7GYAYQJ2P2L6TKTRT4H7V425T5X4FBYPQ6EQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756634764 (525h34m43.627717799s) [noise=true , address=45.91.116.9:4261 , piece_id=SR6L37662MOIO6XI3MDCZMM7MZE3DYVQQWMOIUA4TFKNEW4LJ2IA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756719245 (525h34m43.62780229s) [noise=true , address=145.79.227.35:4482 , piece_id=HHZWUYFEG52B2PH7QRNZ5RM523YHCHF6FC72WGBQ27IQVAAMQVHA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.756993797 (525h34m43.628076843s) [noise=true , address=45.91.116.11:4120 , piece_id=LPLOJJB6FJUDLILTLVQR3PMDWRN2KA4FQ37CZ6NWOVKZQCIYPC3A, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757064191 (525h34m43.628147227s) [noise=true , address=145.79.227.48:4302 , piece_id=UEXBPBPZRDU62WSUQ4YXQBVYXSDTTWS4MYPUL42PUPTCABFATJUA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757305952 (525h34m43.628388987s) [noise=true , address=145.79.227.36:4517 , piece_id=2W4DJXMAPDIWJVDAW3J2RCFVPU7MXXKK6QVMJRUBAJKDDPCQBLXA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757353934 (525h34m43.628436979s) [noise=true , address=145.79.227.25:4617 , piece_id=EZHMCPA4OL3G54BBARBTZVGAEQY5JAG5HJWSEGD2PXZSHJFBVGRA, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757563783 (525h34m43.628646829s) [noise=true , address=145.79.227.50:4373 , piece_id=DDQR5GOC6E2227SPJDN7IOI5XUALTHL2PY73TK2RGT4BY6QRGO4Q, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757655117 (525h34m43.628738152s) [noise=true , address=45.91.116.10:4082 , piece_id=66Z7EITUPKKNXG6DWLEYNCYKW62O4CCNTUEL66UF676DIMTPGNAQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757855238 (525h34m43.628938273s) [noise=true , address=145.79.227.39:4014 , piece_id=QXPAP3YB5ASRU2I66WWX5MLALOFPXIP3QEV7KEVNY2WC7Z6ALZCQ, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:43.757995475 (525h34m43.62907852s) [noise=true , address=145.79.227.49:4339 , piece_id=MEBKJDKVADF6E2DU422PPUE4NAG4XUDCGT7DCQO2NQLJTR76JC2Q, seg_pos={0 0}, upload=461021]: Uploading piece... Dec 27 13:03:45.476103088 (525h34m45.347186133s) [upload=461021]: Next returned nil segment Dec 27 13:03:45.476120642 (525h34m45.347203687s) [upload=461021]: Waiting for error group managing segments... Dec 27 13:03:45.760277926 (525h34m45.631360971s) [noise=true , address=145.79.227.25:4617 , piece_id=EZHMCPA4OL3G54BBARBTZVGAEQY5JAG5HJWSEGD2PXZSHJFBVGRA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.771720365 (525h34m45.6428034s) [noise=true , address=145.79.227.52:4447 , piece_id=UAO2UD75QTHKZM7MOTB2KXWQHDRV4PVUC7W72N7QUEDMU6L6CRRQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.777751239 (525h34m45.648834284s) [noise=true , address=145.79.227.39:4014 , piece_id=QXPAP3YB5ASRU2I66WWX5MLALOFPXIP3QEV7KEVNY2WC7Z6ALZCQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.777787488 (525h34m45.648870523s) [noise=true , address=145.79.227.41:4085 , piece_id=DUIWCMBHKUTQAR75TPLWL7C2TWZQPF4MU7UW6AQUVIBTZ7K7NIXA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.781443319 (525h34m45.652526354s) [noise=true , address=145.79.227.49:4339 , piece_id=MEBKJDKVADF6E2DU422PPUE4NAG4XUDCGT7DCQO2NQLJTR76JC2Q, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.781702642 (525h34m45.652785677s) [noise=true , address=145.79.227.48:4302 , piece_id=UEXBPBPZRDU62WSUQ4YXQBVYXSDTTWS4MYPUL42PUPTCABFATJUA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.783234930 (525h34m45.654317975s) [noise=true , address=145.79.227.50:4373 , piece_id=DDQR5GOC6E2227SPJDN7IOI5XUALTHL2PY73TK2RGT4BY6QRGO4Q, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.788660451 (525h34m45.659743486s) [noise=true , address=145.79.227.36:4517 , piece_id=2W4DJXMAPDIWJVDAW3J2RCFVPU7MXXKK6QVMJRUBAJKDDPCQBLXA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.792474123 (525h34m45.663557158s) [noise=true , address=145.79.227.35:4482 , piece_id=HHZWUYFEG52B2PH7QRNZ5RM523YHCHF6FC72WGBQ27IQVAAMQVHA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.996308238 (525h34m45.867391274s) [noise=true , address=45.91.116.7:4012 , piece_id=LI37CDKQFPU3QY2VG2FM7OWRUMQMJHELZTDTAAKXAE7JVXULPV3Q, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.996693712 (525h34m45.867776747s) [noise=true , address=45.91.116.5:4299 , piece_id=DTVGF2DYXRTNTGDZZOF5IQE5K4TDZORYV4AABTDHMR4D65IEWAMQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:45.997658259 (525h34m45.868741294s) [noise=true , address=45.91.116.10:4082 , piece_id=66Z7EITUPKKNXG6DWLEYNCYKW62O4CCNTUEL66UF676DIMTPGNAQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:46.021556505 (525h34m45.89263955s) [noise=true , address=45.91.116.11:4120 , piece_id=LPLOJJB6FJUDLILTLVQR3PMDWRN2KA4FQ37CZ6NWOVKZQCIYPC3A, seg_pos={0 0}, upload=461021]: Done uploading piece. err: <nil> Dec 27 13:03:46.021578117 (525h34m45.892661162s) [seg_pos={0 0}, upload=461021]: Segment reached optimal threshold of 13 pieces. Dec 27 13:03:46.021758440 (525h34m45.892841475s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZMQ5Y7NRQNAIWO5M6MV7BMOUWHXCGC2C4NPSGGBHHBTUOROCUZQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 13:03:46.021765012 (525h34m45.892848047s) [noise=true , address=45.91.116.14:4190 , piece_id=5ZMQ5Y7NRQNAIWO5M6MV7BMOUWHXCGC2C4NPSGGBHHBTUOROCUZQ, seg_pos={0 0}, upload=461021]: Long tail cancellation due to optimal threshold Dec 27 13:03:46.021990362 (525h34m45.893073397s) [noise=true , address=45.91.116.9:4261 , piece_id=SR6L37662MOIO6XI3MDCZMM7MZE3DYVQQWMOIUA4TFKNEW4LJ2IA, seg_pos={0 0}, upload=461021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:37312->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:37312->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37312->45.91.116.9:4261: use of closed network connection Dec 27 13:03:46.021997916 (525h34m45.893080961s) [noise=true , address=45.91.116.9:4261 , piece_id=SR6L37662MOIO6XI3MDCZMM7MZE3DYVQQWMOIUA4TFKNEW4LJ2IA, seg_pos={0 0}, upload=461021]: Long tail cancellation due to optimal threshold Dec 27 13:03:46.022004799 (525h34m45.893087834s) [noise=true , address=77.90.31.10:4519 , piece_id=ANQLWAXPRZAM3YNC7GYAYQJ2P2L6TKTRT4H7V425T5X4FBYPQ6EQ, seg_pos={0 0}, upload=461021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:57404->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:57404->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57404->77.90.31.10:4519: use of closed network connection Dec 27 13:03:46.022011151 (525h34m45.893094196s) [noise=true , address=77.90.31.10:4519 , piece_id=ANQLWAXPRZAM3YNC7GYAYQJ2P2L6TKTRT4H7V425T5X4FBYPQ6EQ, seg_pos={0 0}, upload=461021]: Long tail cancellation due to optimal threshold Dec 27 13:03:46.022168371 (525h34m45.893251406s) [seg_pos={0 0}, upload=461021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 13:03:46.022237903 (525h34m45.893320938s) [upload=461021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 13:03:46.022241891 (525h34m45.893324936s) [upload=461021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 13:03:46.317533958 (525h34m46.188616993s) [upload=461021]: Done uploading segments... Dec 27 13:03:46.317554166 (525h34m46.188637201s) [upload=461021]: Upload finished. err: <nil> Duration: 30.004457648s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...........X...........X.X......X..............X..X.X...X.XX] duration: 5.001747728s trace: https://jaeger.dev.storj.io/trace/27eb7c025fd2e151 FAIL (2025-12-27 13:03:43 UTC) orbiter warning: context deadline exceeded Duration: 5.0017743s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [............X...........X.X......X..............X..X.X...X.X] duration: 5.001359601s trace: https://jaeger.dev.storj.io/trace/369e90c12d29e7f2 FAIL (2025-12-27 13:03:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001384258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X................................X.....................] duration: 39.529140543s trace: https://jaeger.dev.storj.io/trace/67980cd8fff84f06 FAIL (2025-12-27 13:02:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sy3iQ81wRjoqXs8RhuZW677SBDn4VP6tw14hbozoWenFfgdqu, Piece ID: 5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA) context deadline exceeded; piecestore: (Node ID: 1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, Piece ID: 5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA) context deadline exceeded; piecestore: (Node ID: 124KqWoyeyoKbtoEzbWBuRa9owqLiUsfNSp89oa2fEpJCRZz4Ye, Piece ID: KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA) context deadline exceeded ======LOGS====== 13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461479694 (525h13m42.332562739s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.461737203 (525h13m42.332820239s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462003020 (525h13m42.333086055s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462034240 (525h13m42.333117285s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462525545 (525h13m42.33360859s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462739873 (525h13m42.333822918s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.462797022 (525h13m42.333880057s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463073168 (525h13m42.334156203s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463329756 (525h13m42.334412801s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463569683 (525h13m42.334652728s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463635078 (525h13m42.334718123s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.463923096 (525h13m42.335006161s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.464270568 (525h13m42.335353603s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:42.467068917 (525h13m42.338151962s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Uploading piece... Dec 27 12:42:43.771082828 (525h13m43.642165873s) [upload=460711]: Next returned nil segment Dec 27 12:42:43.771096956 (525h13m43.642179991s) [upload=460711]: Waiting for error group managing segments... Dec 27 12:42:44.300665304 (525h13m44.171748339s) [noise=true , address=193.149.164.143:4701 , piece_id=WF3J6WSZC6NANRK4YAWGYQBV2ST4OTELNHLWQD5UKJENDGIL2VGQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.334817064 (525h13m44.205900099s) [noise=true , address=145.79.227.10:4429 , piece_id=P5B2E36C6FIIDADPFACL7TBNPAXQAA35ZKBS4RFLWXQQDLTFOT7Q, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.357754701 (525h13m44.228837736s) [noise=true , address=145.79.227.43:4186 , piece_id=UMBWSDS33EJ3NCWJR3IBSIH34JCXTIBOPCRIM3NAUHRS4JSVVSNA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.370947372 (525h13m44.242030417s) [noise=true , address=145.79.227.16:4932 , piece_id=U32GKH6ZMTKVKLKS3726GKLKRLGRWK4CZEDK5OLIGFQNJOHHWBYA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.544213540 (525h13m44.415296585s) [noise=true , address=77.90.31.13:4630 , piece_id=4IAUTTBMXUUUMRNJ5YOWKLOHNMLJKNC53VQXGF3MS443N654PWOQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.648013103 (525h13m44.519096138s) [noise=true , address=77.90.31.11:4570 , piece_id=6BP3FAA6SZ7CMRBV3DWLDTXPE6IZ4ZD5T6CPQZ2UEKVCU4SFNF5A, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.701365715 (525h13m44.57244876s) [noise=true , address=45.91.116.12:4259 , piece_id=KZYRFWAEQID4HBRDKT2IAH2435URUPFGA4PDJFGTL6LXQ774HRUA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:44.731017397 (525h13m44.602100442s) [noise=true , address=77.90.31.21:4911 , piece_id=XEXTMEJTSNLQAWKNUJ55EL7FTJF5ZM5I5G5QMLYWO44RQWS3CMYQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.184136271 (525h13m45.055219306s) [noise=true , address=77.90.31.14:4669 , piece_id=GJCSWCJQ5ROQLVAO4UCFUZUSYSBJTDWHLEDTSDEKRFTJJDPC43VQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.305192633 (525h13m45.176275678s) [noise=true , address=77.90.31.19:4845 , piece_id=5GCXBSMKCCNDRUTG2DSJIMCH6T2IRSXOV4ANBBF7GLDSEIGJISTA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.331752207 (525h13m45.202835252s) [noise=true , address=145.79.227.44:4225 , piece_id=ONXUARISWCV77VJVBSC77RZJKGHGOM7BEPUPOB5KMVQMQLUW26RQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.337387216 (525h13m45.208470251s) [noise=true , address=45.91.116.13:4164 , piece_id=5AG5XYJCCTTZKAG6KJVAS24IKAIBQRVV7EYSJFFOLPMKRTR7HVJA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.400681550 (525h13m45.271764595s) [noise=true , address=77.90.31.15:4726 , piece_id=G7GSYFL2D5I34HRSA4D3IHMHGDRPDPQW67EYIMPRVJVUSRRLWDWQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.548984854 (525h13m45.420067899s) [noise=true , address=145.79.227.40:4053 , piece_id=QELWG667HQ62XDYNITDKYPS26UKRHNYNNMCX7WLAKWAEY4L3HGLA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: <nil> Dec 27 12:42:45.549007277 (525h13m45.420090312s) [seg_pos={0 0}, upload=460711]: Segment reached optimal threshold of 14 pieces. Dec 27 12:42:45.549218289 (525h13m45.420301334s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549228118 (525h13m45.420311153s) [noise=true , address=77.90.31.16:4756 , piece_id=WBJIXD3POPLHRU2BM7OQOXV2NVPRVQLN6VK75TVC6JLG7FSKK3WA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549354449 (525h13m45.420437494s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14ZdgenD4m2bMMmaHVyK3YivrzdSnrp72JvMJsk1uecW6KKFjn): protocol: expected piece hash; context canceled; EOF Dec 27 12:42:45.549418640 (525h13m45.420501685s) [noise=true , address=145.79.227.34:4925 , piece_id=FFWY3U5DJLFQUOUDYP2TVJ5U7ETWSRDDGSDBX4QLMBPIFZI2SPSQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549636385 (525h13m45.42071942s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42058->64.49.13.169:4711: use of closed network connection Dec 27 12:42:45.549643619 (525h13m45.420726664s) [noise=true , address=64.49.13.169:4711 , piece_id=VQLFSLY3PZNSP2UPDHQL6JCB7JUYFJPCDQP5E5FCGXJNQCZB5CEQ, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549904335 (525h13m45.42098738s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48902->77.90.31.18:4814: use of closed network connection Dec 27 12:42:45.549913032 (525h13m45.420996067s) [noise=true , address=77.90.31.18:4814 , piece_id=N54VE2GAA7HUHG66WO2OYQKGSCH34OZJZTJE7VWJ4DX4EDM2FLEA, seg_pos={0 0}, upload=460711]: Long tail cancellation due to optimal threshold Dec 27 12:42:45.549933541 (525h13m45.421016576s) [seg_pos={0 0}, upload=460711]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:42:45.550005027 (525h13m45.421088062s) [upload=460711]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:42:45.550009655 (525h13m45.42109269s) [upload=460711]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:42:45.852172915 (525h13m45.72325596s) [upload=460711]: Done uploading segments... Dec 27 12:42:45.852195298 (525h13m45.723278343s) [upload=460711]: Upload finished. err: <nil> Duration: 39.529160391s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)