Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 15.847845492s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:19:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 15.847877765s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.744075944s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 16:18:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 6.744103908s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 21.863468878s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:18:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 21.863498933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.969707402s trace: https://jaeger.dev.storj.io/trace/2a193e7abaa3c1dd FAIL (2025-12-18 16:18:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 16:11:04.958723652 (312h42m4.829806697s) [upload=274190]: Starting upload Dec 18 16:11:04.958798045 (312h42m4.82988109s) [upload=274190]: Done starting upload Dec 18 16:11:04.958868659 (312h42m4.829951704s) [upload=274190]: Uploading segments... Dec 18 16:11:04.959025928 (312h42m4.830108963s) [upload=274190]: Got next segment. Inline: false Dec 18 16:11:04.959102023 (312h42m4.830185068s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 16:11:04.959110159 (312h42m4.830193214s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 16:11:05.200556701 (312h42m5.071639746s) [seg_pos={0 0}, upload=274190]: Begin upload segment... Dec 18 16:11:05.200604913 (312h42m5.071687948s) [seg_pos={0 0}, upload=274190]: Done begin upload segment. Dec 18 16:11:05.200632004 (312h42m5.071715039s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.200687510 (312h42m5.071770575s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201175339 (312h42m5.072258384s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201704236 (312h42m5.072787291s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202236611 (312h42m5.073319656s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202746200 (312h42m5.073829255s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203240522 (312h42m5.074323567s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203751084 (312h42m5.074834129s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205036271 (312h42m5.076119306s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205393542 (312h42m5.076476587s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.206472968 (312h42m5.077556013s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.208083484 (312h42m5.079166529s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.209348583 (312h42m5.080431628s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.211021067 (312h42m5.082104112s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.212092929 (312h42m5.083175974s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.213431809 (312h42m5.084514854s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:06.352232932 (312h42m6.223315977s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 18 16:11:06.910606677 (312h42m6.781689722s) [upload=274190]: Next returned nil segment Dec 18 16:11:06.910621466 (312h42m6.781704511s) [upload=274190]: Waiting for error group managing segments... Dec 18 16:11:07.196606158 (312h42m7.067689203s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.197771416 (312h42m7.068854461s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199147727 (312h42m7.070230772s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199886674 (312h42m7.070969719s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.204748963 (312h42m7.075831998s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.211093014 (312h42m7.082176049s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.222671160 (312h42m7.093754205s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.227728892 (312h42m7.098811937s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.228294598 (312h42m7.099377644s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.274781509 (312h42m7.145864544s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.282847367 (312h42m7.153930413s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.447712865 (312h42m7.31879591s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815760922 (312h42m7.686843967s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815788705 (312h42m7.68687175s) [seg_pos={0 0}, upload=274190]: Segment reached optimal threshold of 13 pieces. Dec 18 16:11:07.816001600 (312h42m7.687084645s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection Dec 18 16:11:07.816008693 (312h42m7.687091739s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816127068 (312h42m7.687210114s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection Dec 18 16:11:07.816169560 (312h42m7.687252625s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816209366 (312h42m7.687292411s) [seg_pos={0 0}, upload=274190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 16:11:07.816354492 (312h42m7.687437547s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 16:11:07.816362497 (312h42m7.687445542s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 16:11:08.111101419 (312h42m7.982184454s) [upload=274190]: Done uploading segments... Dec 18 16:11:08.111140293 (312h42m7.982223338s) [upload=274190]: Upload finished. err: <nil> Duration: 5.969728062s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X..X........X...........................................] duration: 3.342627851s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:18:20 UTC) orbiter warning: context deadline exceeded Duration: 3.342659119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 18.353680204s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 16:18:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 18.353709821s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 8.972337158s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:18:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 8.972370011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.009950666s trace: https://jaeger.dev.storj.io/trace/2a193e7abaa3c1dd FAIL (2025-12-18 16:17:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 16:11:04.958723652 (312h42m4.829806697s) [upload=274190]: Starting upload Dec 18 16:11:04.958798045 (312h42m4.82988109s) [upload=274190]: Done starting upload Dec 18 16:11:04.958868659 (312h42m4.829951704s) [upload=274190]: Uploading segments... Dec 18 16:11:04.959025928 (312h42m4.830108963s) [upload=274190]: Got next segment. Inline: false Dec 18 16:11:04.959102023 (312h42m4.830185068s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 16:11:04.959110159 (312h42m4.830193214s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 16:11:05.200556701 (312h42m5.071639746s) [seg_pos={0 0}, upload=274190]: Begin upload segment... Dec 18 16:11:05.200604913 (312h42m5.071687948s) [seg_pos={0 0}, upload=274190]: Done begin upload segment. Dec 18 16:11:05.200632004 (312h42m5.071715039s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.200687510 (312h42m5.071770575s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201175339 (312h42m5.072258384s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201704236 (312h42m5.072787291s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202236611 (312h42m5.073319656s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202746200 (312h42m5.073829255s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203240522 (312h42m5.074323567s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203751084 (312h42m5.074834129s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205036271 (312h42m5.076119306s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205393542 (312h42m5.076476587s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.206472968 (312h42m5.077556013s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.208083484 (312h42m5.079166529s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.209348583 (312h42m5.080431628s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.211021067 (312h42m5.082104112s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.212092929 (312h42m5.083175974s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.213431809 (312h42m5.084514854s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:06.352232932 (312h42m6.223315977s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 18 16:11:06.910606677 (312h42m6.781689722s) [upload=274190]: Next returned nil segment Dec 18 16:11:06.910621466 (312h42m6.781704511s) [upload=274190]: Waiting for error group managing segments... Dec 18 16:11:07.196606158 (312h42m7.067689203s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.197771416 (312h42m7.068854461s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199147727 (312h42m7.070230772s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199886674 (312h42m7.070969719s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.204748963 (312h42m7.075831998s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.211093014 (312h42m7.082176049s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.222671160 (312h42m7.093754205s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.227728892 (312h42m7.098811937s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.228294598 (312h42m7.099377644s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.274781509 (312h42m7.145864544s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.282847367 (312h42m7.153930413s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.447712865 (312h42m7.31879591s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815760922 (312h42m7.686843967s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815788705 (312h42m7.68687175s) [seg_pos={0 0}, upload=274190]: Segment reached optimal threshold of 13 pieces. Dec 18 16:11:07.816001600 (312h42m7.687084645s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection Dec 18 16:11:07.816008693 (312h42m7.687091739s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816127068 (312h42m7.687210114s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection Dec 18 16:11:07.816169560 (312h42m7.687252625s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816209366 (312h42m7.687292411s) [seg_pos={0 0}, upload=274190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 16:11:07.816354492 (312h42m7.687437547s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 16:11:07.816362497 (312h42m7.687445542s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 16:11:08.111101419 (312h42m7.982184454s) [upload=274190]: Done uploading segments... Dec 18 16:11:08.111140293 (312h42m7.982223338s) [upload=274190]: Upload finished. err: <nil> Duration: 6.009987125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 21.942826651s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:17:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 21.942854375s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X..X........X..........................................] duration: 3.633978463s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:17:07 UTC) orbiter warning: context deadline exceeded Duration: 3.634111227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 8.893427898s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 16:16:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 8.8934544s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 14.427663794s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:16:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 14.42769333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 17.154208777s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:16:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 17.154238424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.965276183s trace: https://jaeger.dev.storj.io/trace/2a193e7abaa3c1dd FAIL (2025-12-18 16:16:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 16:11:04.958723652 (312h42m4.829806697s) [upload=274190]: Starting upload Dec 18 16:11:04.958798045 (312h42m4.82988109s) [upload=274190]: Done starting upload Dec 18 16:11:04.958868659 (312h42m4.829951704s) [upload=274190]: Uploading segments... Dec 18 16:11:04.959025928 (312h42m4.830108963s) [upload=274190]: Got next segment. Inline: false Dec 18 16:11:04.959102023 (312h42m4.830185068s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 16:11:04.959110159 (312h42m4.830193214s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 16:11:05.200556701 (312h42m5.071639746s) [seg_pos={0 0}, upload=274190]: Begin upload segment... Dec 18 16:11:05.200604913 (312h42m5.071687948s) [seg_pos={0 0}, upload=274190]: Done begin upload segment. Dec 18 16:11:05.200632004 (312h42m5.071715039s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.200687510 (312h42m5.071770575s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201175339 (312h42m5.072258384s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201704236 (312h42m5.072787291s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202236611 (312h42m5.073319656s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202746200 (312h42m5.073829255s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203240522 (312h42m5.074323567s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203751084 (312h42m5.074834129s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205036271 (312h42m5.076119306s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205393542 (312h42m5.076476587s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.206472968 (312h42m5.077556013s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.208083484 (312h42m5.079166529s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.209348583 (312h42m5.080431628s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.211021067 (312h42m5.082104112s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.212092929 (312h42m5.083175974s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.213431809 (312h42m5.084514854s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:06.352232932 (312h42m6.223315977s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 18 16:11:06.910606677 (312h42m6.781689722s) [upload=274190]: Next returned nil segment Dec 18 16:11:06.910621466 (312h42m6.781704511s) [upload=274190]: Waiting for error group managing segments... Dec 18 16:11:07.196606158 (312h42m7.067689203s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.197771416 (312h42m7.068854461s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199147727 (312h42m7.070230772s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199886674 (312h42m7.070969719s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.204748963 (312h42m7.075831998s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.211093014 (312h42m7.082176049s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.222671160 (312h42m7.093754205s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.227728892 (312h42m7.098811937s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.228294598 (312h42m7.099377644s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.274781509 (312h42m7.145864544s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.282847367 (312h42m7.153930413s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.447712865 (312h42m7.31879591s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815760922 (312h42m7.686843967s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815788705 (312h42m7.68687175s) [seg_pos={0 0}, upload=274190]: Segment reached optimal threshold of 13 pieces. Dec 18 16:11:07.816001600 (312h42m7.687084645s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection Dec 18 16:11:07.816008693 (312h42m7.687091739s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816127068 (312h42m7.687210114s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection Dec 18 16:11:07.816169560 (312h42m7.687252625s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816209366 (312h42m7.687292411s) [seg_pos={0 0}, upload=274190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 16:11:07.816354492 (312h42m7.687437547s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 16:11:07.816362497 (312h42m7.687445542s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 16:11:08.111101419 (312h42m7.982184454s) [upload=274190]: Done uploading segments... Dec 18 16:11:08.111140293 (312h42m7.982223338s) [upload=274190]: Upload finished. err: <nil> Duration: 5.965306972s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 18.611472192s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:16:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 18.611498153s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.....X..X........X.........................................] duration: 3.581340597s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:16:03 UTC) orbiter warning: context deadline exceeded Duration: 3.581411531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 10.290078911s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 16:15:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 10.290172711s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X.....X..X........X........................................] duration: 4.048554572s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:15:46 UTC) orbiter warning: context deadline exceeded Duration: 4.048660555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.979217781s trace: https://jaeger.dev.storj.io/trace/2a193e7abaa3c1dd FAIL (2025-12-18 16:15:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 16:11:04.958723652 (312h42m4.829806697s) [upload=274190]: Starting upload Dec 18 16:11:04.958798045 (312h42m4.82988109s) [upload=274190]: Done starting upload Dec 18 16:11:04.958868659 (312h42m4.829951704s) [upload=274190]: Uploading segments... Dec 18 16:11:04.959025928 (312h42m4.830108963s) [upload=274190]: Got next segment. Inline: false Dec 18 16:11:04.959102023 (312h42m4.830185068s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 16:11:04.959110159 (312h42m4.830193214s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 16:11:05.200556701 (312h42m5.071639746s) [seg_pos={0 0}, upload=274190]: Begin upload segment... Dec 18 16:11:05.200604913 (312h42m5.071687948s) [seg_pos={0 0}, upload=274190]: Done begin upload segment. Dec 18 16:11:05.200632004 (312h42m5.071715039s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.200687510 (312h42m5.071770575s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201175339 (312h42m5.072258384s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201704236 (312h42m5.072787291s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202236611 (312h42m5.073319656s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202746200 (312h42m5.073829255s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203240522 (312h42m5.074323567s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203751084 (312h42m5.074834129s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205036271 (312h42m5.076119306s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205393542 (312h42m5.076476587s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.206472968 (312h42m5.077556013s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.208083484 (312h42m5.079166529s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.209348583 (312h42m5.080431628s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.211021067 (312h42m5.082104112s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.212092929 (312h42m5.083175974s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.213431809 (312h42m5.084514854s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:06.352232932 (312h42m6.223315977s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 18 16:11:06.910606677 (312h42m6.781689722s) [upload=274190]: Next returned nil segment Dec 18 16:11:06.910621466 (312h42m6.781704511s) [upload=274190]: Waiting for error group managing segments... Dec 18 16:11:07.196606158 (312h42m7.067689203s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.197771416 (312h42m7.068854461s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199147727 (312h42m7.070230772s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199886674 (312h42m7.070969719s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.204748963 (312h42m7.075831998s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.211093014 (312h42m7.082176049s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.222671160 (312h42m7.093754205s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.227728892 (312h42m7.098811937s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.228294598 (312h42m7.099377644s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.274781509 (312h42m7.145864544s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.282847367 (312h42m7.153930413s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.447712865 (312h42m7.31879591s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815760922 (312h42m7.686843967s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815788705 (312h42m7.68687175s) [seg_pos={0 0}, upload=274190]: Segment reached optimal threshold of 13 pieces. Dec 18 16:11:07.816001600 (312h42m7.687084645s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection Dec 18 16:11:07.816008693 (312h42m7.687091739s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816127068 (312h42m7.687210114s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection Dec 18 16:11:07.816169560 (312h42m7.687252625s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816209366 (312h42m7.687292411s) [seg_pos={0 0}, upload=274190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 16:11:07.816354492 (312h42m7.687437547s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 16:11:07.816362497 (312h42m7.687445542s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 16:11:08.111101419 (312h42m7.982184454s) [upload=274190]: Done uploading segments... Dec 18 16:11:08.111140293 (312h42m7.982223338s) [upload=274190]: Upload finished. err: <nil> Duration: 5.979265922s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X.....X..X........X.......................................] duration: 3.368382709s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:14:59 UTC) orbiter warning: context deadline exceeded Duration: 3.368399472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 7.804326762s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 16:14:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 7.804355867s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.980871618s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:14:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 6.980898879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.81947391s trace: https://jaeger.dev.storj.io/trace/2a193e7abaa3c1dd FAIL (2025-12-18 16:14:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 16:11:04.958723652 (312h42m4.829806697s) [upload=274190]: Starting upload Dec 18 16:11:04.958798045 (312h42m4.82988109s) [upload=274190]: Done starting upload Dec 18 16:11:04.958868659 (312h42m4.829951704s) [upload=274190]: Uploading segments... Dec 18 16:11:04.959025928 (312h42m4.830108963s) [upload=274190]: Got next segment. Inline: false Dec 18 16:11:04.959102023 (312h42m4.830185068s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 16:11:04.959110159 (312h42m4.830193214s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 16:11:05.200556701 (312h42m5.071639746s) [seg_pos={0 0}, upload=274190]: Begin upload segment... Dec 18 16:11:05.200604913 (312h42m5.071687948s) [seg_pos={0 0}, upload=274190]: Done begin upload segment. Dec 18 16:11:05.200632004 (312h42m5.071715039s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.200687510 (312h42m5.071770575s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201175339 (312h42m5.072258384s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201704236 (312h42m5.072787291s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202236611 (312h42m5.073319656s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202746200 (312h42m5.073829255s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203240522 (312h42m5.074323567s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203751084 (312h42m5.074834129s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205036271 (312h42m5.076119306s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205393542 (312h42m5.076476587s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.206472968 (312h42m5.077556013s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.208083484 (312h42m5.079166529s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.209348583 (312h42m5.080431628s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.211021067 (312h42m5.082104112s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.212092929 (312h42m5.083175974s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.213431809 (312h42m5.084514854s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:06.352232932 (312h42m6.223315977s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 18 16:11:06.910606677 (312h42m6.781689722s) [upload=274190]: Next returned nil segment Dec 18 16:11:06.910621466 (312h42m6.781704511s) [upload=274190]: Waiting for error group managing segments... Dec 18 16:11:07.196606158 (312h42m7.067689203s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.197771416 (312h42m7.068854461s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199147727 (312h42m7.070230772s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199886674 (312h42m7.070969719s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.204748963 (312h42m7.075831998s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.211093014 (312h42m7.082176049s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.222671160 (312h42m7.093754205s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.227728892 (312h42m7.098811937s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.228294598 (312h42m7.099377644s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.274781509 (312h42m7.145864544s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.282847367 (312h42m7.153930413s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.447712865 (312h42m7.31879591s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815760922 (312h42m7.686843967s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815788705 (312h42m7.68687175s) [seg_pos={0 0}, upload=274190]: Segment reached optimal threshold of 13 pieces. Dec 18 16:11:07.816001600 (312h42m7.687084645s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection Dec 18 16:11:07.816008693 (312h42m7.687091739s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816127068 (312h42m7.687210114s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection Dec 18 16:11:07.816169560 (312h42m7.687252625s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816209366 (312h42m7.687292411s) [seg_pos={0 0}, upload=274190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 16:11:07.816354492 (312h42m7.687437547s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 16:11:07.816362497 (312h42m7.687445542s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 16:11:08.111101419 (312h42m7.982184454s) [upload=274190]: Done uploading segments... Dec 18 16:11:08.111140293 (312h42m7.982223338s) [upload=274190]: Upload finished. err: <nil> Duration: 5.819497756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 15.277282698s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:13:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 15.277307965s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.074231154s trace: https://jaeger.dev.storj.io/trace/2a193e7abaa3c1dd FAIL (2025-12-18 16:13:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 16:11:04.958723652 (312h42m4.829806697s) [upload=274190]: Starting upload Dec 18 16:11:04.958798045 (312h42m4.82988109s) [upload=274190]: Done starting upload Dec 18 16:11:04.958868659 (312h42m4.829951704s) [upload=274190]: Uploading segments... Dec 18 16:11:04.959025928 (312h42m4.830108963s) [upload=274190]: Got next segment. Inline: false Dec 18 16:11:04.959102023 (312h42m4.830185068s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 16:11:04.959110159 (312h42m4.830193214s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 16:11:05.200556701 (312h42m5.071639746s) [seg_pos={0 0}, upload=274190]: Begin upload segment... Dec 18 16:11:05.200604913 (312h42m5.071687948s) [seg_pos={0 0}, upload=274190]: Done begin upload segment. Dec 18 16:11:05.200632004 (312h42m5.071715039s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.200687510 (312h42m5.071770575s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201175339 (312h42m5.072258384s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201704236 (312h42m5.072787291s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202236611 (312h42m5.073319656s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202746200 (312h42m5.073829255s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203240522 (312h42m5.074323567s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203751084 (312h42m5.074834129s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205036271 (312h42m5.076119306s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205393542 (312h42m5.076476587s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.206472968 (312h42m5.077556013s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.208083484 (312h42m5.079166529s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.209348583 (312h42m5.080431628s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.211021067 (312h42m5.082104112s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.212092929 (312h42m5.083175974s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.213431809 (312h42m5.084514854s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:06.352232932 (312h42m6.223315977s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 18 16:11:06.910606677 (312h42m6.781689722s) [upload=274190]: Next returned nil segment Dec 18 16:11:06.910621466 (312h42m6.781704511s) [upload=274190]: Waiting for error group managing segments... Dec 18 16:11:07.196606158 (312h42m7.067689203s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.197771416 (312h42m7.068854461s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199147727 (312h42m7.070230772s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199886674 (312h42m7.070969719s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.204748963 (312h42m7.075831998s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.211093014 (312h42m7.082176049s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.222671160 (312h42m7.093754205s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.227728892 (312h42m7.098811937s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.228294598 (312h42m7.099377644s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.274781509 (312h42m7.145864544s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.282847367 (312h42m7.153930413s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.447712865 (312h42m7.31879591s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815760922 (312h42m7.686843967s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815788705 (312h42m7.68687175s) [seg_pos={0 0}, upload=274190]: Segment reached optimal threshold of 13 pieces. Dec 18 16:11:07.816001600 (312h42m7.687084645s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection Dec 18 16:11:07.816008693 (312h42m7.687091739s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816127068 (312h42m7.687210114s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection Dec 18 16:11:07.816169560 (312h42m7.687252625s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816209366 (312h42m7.687292411s) [seg_pos={0 0}, upload=274190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 16:11:07.816354492 (312h42m7.687437547s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 16:11:07.816362497 (312h42m7.687445542s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 16:11:08.111101419 (312h42m7.982184454s) [upload=274190]: Done uploading segments... Dec 18 16:11:08.111140293 (312h42m7.982223338s) [upload=274190]: Upload finished. err: <nil> Duration: 6.074259749s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.283110543s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 16:13:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 6.283134148s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X.....X..X........X......................................] duration: 3.674834817s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:13:48 UTC) orbiter warning: context deadline exceeded Duration: 3.674889181s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X...X.....X..X........X.....................................] duration: 3.495749768s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:13:06 UTC) orbiter warning: context deadline exceeded Duration: 3.495774042s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 16.884252766s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:13:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 16.884284435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.332432183s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 16:12:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 6.332458073s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.069233402s trace: https://jaeger.dev.storj.io/trace/2a193e7abaa3c1dd FAIL (2025-12-18 16:12:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 16:11:04.958723652 (312h42m4.829806697s) [upload=274190]: Starting upload Dec 18 16:11:04.958798045 (312h42m4.82988109s) [upload=274190]: Done starting upload Dec 18 16:11:04.958868659 (312h42m4.829951704s) [upload=274190]: Uploading segments... Dec 18 16:11:04.959025928 (312h42m4.830108963s) [upload=274190]: Got next segment. Inline: false Dec 18 16:11:04.959102023 (312h42m4.830185068s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 16:11:04.959110159 (312h42m4.830193214s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 16:11:05.200556701 (312h42m5.071639746s) [seg_pos={0 0}, upload=274190]: Begin upload segment... Dec 18 16:11:05.200604913 (312h42m5.071687948s) [seg_pos={0 0}, upload=274190]: Done begin upload segment. Dec 18 16:11:05.200632004 (312h42m5.071715039s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.200687510 (312h42m5.071770575s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201175339 (312h42m5.072258384s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201704236 (312h42m5.072787291s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202236611 (312h42m5.073319656s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202746200 (312h42m5.073829255s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203240522 (312h42m5.074323567s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203751084 (312h42m5.074834129s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205036271 (312h42m5.076119306s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205393542 (312h42m5.076476587s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.206472968 (312h42m5.077556013s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.208083484 (312h42m5.079166529s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.209348583 (312h42m5.080431628s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.211021067 (312h42m5.082104112s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.212092929 (312h42m5.083175974s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.213431809 (312h42m5.084514854s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:06.352232932 (312h42m6.223315977s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 18 16:11:06.910606677 (312h42m6.781689722s) [upload=274190]: Next returned nil segment Dec 18 16:11:06.910621466 (312h42m6.781704511s) [upload=274190]: Waiting for error group managing segments... Dec 18 16:11:07.196606158 (312h42m7.067689203s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.197771416 (312h42m7.068854461s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199147727 (312h42m7.070230772s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199886674 (312h42m7.070969719s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.204748963 (312h42m7.075831998s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.211093014 (312h42m7.082176049s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.222671160 (312h42m7.093754205s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.227728892 (312h42m7.098811937s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.228294598 (312h42m7.099377644s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.274781509 (312h42m7.145864544s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.282847367 (312h42m7.153930413s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.447712865 (312h42m7.31879591s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815760922 (312h42m7.686843967s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815788705 (312h42m7.68687175s) [seg_pos={0 0}, upload=274190]: Segment reached optimal threshold of 13 pieces. Dec 18 16:11:07.816001600 (312h42m7.687084645s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection Dec 18 16:11:07.816008693 (312h42m7.687091739s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816127068 (312h42m7.687210114s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection Dec 18 16:11:07.816169560 (312h42m7.687252625s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816209366 (312h42m7.687292411s) [seg_pos={0 0}, upload=274190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 16:11:07.816354492 (312h42m7.687437547s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 16:11:07.816362497 (312h42m7.687445542s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 16:11:08.111101419 (312h42m7.982184454s) [upload=274190]: Done uploading segments... Dec 18 16:11:08.111140293 (312h42m7.982223338s) [upload=274190]: Upload finished. err: <nil> Duration: 6.069265724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 7.246911511s trace: https://jaeger.dev.storj.io/trace/2a193e7abaa3c1dd FAIL (2025-12-18 16:12:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 16:11:04.958723652 (312h42m4.829806697s) [upload=274190]: Starting upload Dec 18 16:11:04.958798045 (312h42m4.82988109s) [upload=274190]: Done starting upload Dec 18 16:11:04.958868659 (312h42m4.829951704s) [upload=274190]: Uploading segments... Dec 18 16:11:04.959025928 (312h42m4.830108963s) [upload=274190]: Got next segment. Inline: false Dec 18 16:11:04.959102023 (312h42m4.830185068s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 16:11:04.959110159 (312h42m4.830193214s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 16:11:05.200556701 (312h42m5.071639746s) [seg_pos={0 0}, upload=274190]: Begin upload segment... Dec 18 16:11:05.200604913 (312h42m5.071687948s) [seg_pos={0 0}, upload=274190]: Done begin upload segment. Dec 18 16:11:05.200632004 (312h42m5.071715039s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.200687510 (312h42m5.071770575s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201175339 (312h42m5.072258384s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201704236 (312h42m5.072787291s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202236611 (312h42m5.073319656s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202746200 (312h42m5.073829255s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203240522 (312h42m5.074323567s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203751084 (312h42m5.074834129s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205036271 (312h42m5.076119306s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205393542 (312h42m5.076476587s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.206472968 (312h42m5.077556013s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.208083484 (312h42m5.079166529s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.209348583 (312h42m5.080431628s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.211021067 (312h42m5.082104112s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.212092929 (312h42m5.083175974s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.213431809 (312h42m5.084514854s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:06.352232932 (312h42m6.223315977s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 18 16:11:06.910606677 (312h42m6.781689722s) [upload=274190]: Next returned nil segment Dec 18 16:11:06.910621466 (312h42m6.781704511s) [upload=274190]: Waiting for error group managing segments... Dec 18 16:11:07.196606158 (312h42m7.067689203s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.197771416 (312h42m7.068854461s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199147727 (312h42m7.070230772s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199886674 (312h42m7.070969719s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.204748963 (312h42m7.075831998s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.211093014 (312h42m7.082176049s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.222671160 (312h42m7.093754205s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.227728892 (312h42m7.098811937s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.228294598 (312h42m7.099377644s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.274781509 (312h42m7.145864544s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.282847367 (312h42m7.153930413s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.447712865 (312h42m7.31879591s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815760922 (312h42m7.686843967s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815788705 (312h42m7.68687175s) [seg_pos={0 0}, upload=274190]: Segment reached optimal threshold of 13 pieces. Dec 18 16:11:07.816001600 (312h42m7.687084645s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection Dec 18 16:11:07.816008693 (312h42m7.687091739s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816127068 (312h42m7.687210114s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection Dec 18 16:11:07.816169560 (312h42m7.687252625s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816209366 (312h42m7.687292411s) [seg_pos={0 0}, upload=274190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 16:11:07.816354492 (312h42m7.687437547s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 16:11:07.816362497 (312h42m7.687445542s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 16:11:08.111101419 (312h42m7.982184454s) [upload=274190]: Done uploading segments... Dec 18 16:11:08.111140293 (312h42m7.982223338s) [upload=274190]: Upload finished. err: <nil> Duration: 7.246945318s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 7.554619831s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 16:12:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 7.554648865s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X...X.....X..X........X....................................] duration: 3.649275105s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:12:00 UTC) orbiter warning: context deadline exceeded Duration: 3.64930376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 10.268932208s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:11:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 10.268958579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.077091943s trace: https://jaeger.dev.storj.io/trace/2a193e7abaa3c1dd FAIL (2025-12-18 16:11:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 16:11:04.958723652 (312h42m4.829806697s) [upload=274190]: Starting upload Dec 18 16:11:04.958798045 (312h42m4.82988109s) [upload=274190]: Done starting upload Dec 18 16:11:04.958868659 (312h42m4.829951704s) [upload=274190]: Uploading segments... Dec 18 16:11:04.959025928 (312h42m4.830108963s) [upload=274190]: Got next segment. Inline: false Dec 18 16:11:04.959102023 (312h42m4.830185068s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 16:11:04.959110159 (312h42m4.830193214s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 16:11:05.200556701 (312h42m5.071639746s) [seg_pos={0 0}, upload=274190]: Begin upload segment... Dec 18 16:11:05.200604913 (312h42m5.071687948s) [seg_pos={0 0}, upload=274190]: Done begin upload segment. Dec 18 16:11:05.200632004 (312h42m5.071715039s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.200687510 (312h42m5.071770575s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201175339 (312h42m5.072258384s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201704236 (312h42m5.072787291s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202236611 (312h42m5.073319656s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202746200 (312h42m5.073829255s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203240522 (312h42m5.074323567s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203751084 (312h42m5.074834129s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205036271 (312h42m5.076119306s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205393542 (312h42m5.076476587s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.206472968 (312h42m5.077556013s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.208083484 (312h42m5.079166529s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.209348583 (312h42m5.080431628s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.211021067 (312h42m5.082104112s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.212092929 (312h42m5.083175974s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.213431809 (312h42m5.084514854s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:06.352232932 (312h42m6.223315977s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 18 16:11:06.910606677 (312h42m6.781689722s) [upload=274190]: Next returned nil segment Dec 18 16:11:06.910621466 (312h42m6.781704511s) [upload=274190]: Waiting for error group managing segments... Dec 18 16:11:07.196606158 (312h42m7.067689203s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.197771416 (312h42m7.068854461s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199147727 (312h42m7.070230772s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199886674 (312h42m7.070969719s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.204748963 (312h42m7.075831998s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.211093014 (312h42m7.082176049s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.222671160 (312h42m7.093754205s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.227728892 (312h42m7.098811937s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.228294598 (312h42m7.099377644s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.274781509 (312h42m7.145864544s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.282847367 (312h42m7.153930413s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.447712865 (312h42m7.31879591s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815760922 (312h42m7.686843967s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815788705 (312h42m7.68687175s) [seg_pos={0 0}, upload=274190]: Segment reached optimal threshold of 13 pieces. Dec 18 16:11:07.816001600 (312h42m7.687084645s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection Dec 18 16:11:07.816008693 (312h42m7.687091739s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816127068 (312h42m7.687210114s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection Dec 18 16:11:07.816169560 (312h42m7.687252625s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816209366 (312h42m7.687292411s) [seg_pos={0 0}, upload=274190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 16:11:07.816354492 (312h42m7.687437547s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 16:11:07.816362497 (312h42m7.687445542s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 16:11:08.111101419 (312h42m7.982184454s) [upload=274190]: Done uploading segments... Dec 18 16:11:08.111140293 (312h42m7.982223338s) [upload=274190]: Upload finished. err: <nil> Duration: 6.077115217s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003164385s trace: https://jaeger.dev.storj.io/trace/2a193e7abaa3c1dd FAIL (2025-12-18 16:11:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 16:11:04.958723652 (312h42m4.829806697s) [upload=274190]: Starting upload Dec 18 16:11:04.958798045 (312h42m4.82988109s) [upload=274190]: Done starting upload Dec 18 16:11:04.958868659 (312h42m4.829951704s) [upload=274190]: Uploading segments... Dec 18 16:11:04.959025928 (312h42m4.830108963s) [upload=274190]: Got next segment. Inline: false Dec 18 16:11:04.959102023 (312h42m4.830185068s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 16:11:04.959110159 (312h42m4.830193214s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 16:11:05.200556701 (312h42m5.071639746s) [seg_pos={0 0}, upload=274190]: Begin upload segment... Dec 18 16:11:05.200604913 (312h42m5.071687948s) [seg_pos={0 0}, upload=274190]: Done begin upload segment. Dec 18 16:11:05.200632004 (312h42m5.071715039s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.200687510 (312h42m5.071770575s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201175339 (312h42m5.072258384s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.201704236 (312h42m5.072787291s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202236611 (312h42m5.073319656s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.202746200 (312h42m5.073829255s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203240522 (312h42m5.074323567s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.203751084 (312h42m5.074834129s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205036271 (312h42m5.076119306s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.205393542 (312h42m5.076476587s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.206472968 (312h42m5.077556013s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.208083484 (312h42m5.079166529s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.209348583 (312h42m5.080431628s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.211021067 (312h42m5.082104112s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.212092929 (312h42m5.083175974s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:05.213431809 (312h42m5.084514854s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Uploading piece... Dec 18 16:11:06.352232932 (312h42m6.223315977s) [noise=true , address=77.90.31.21:4878 , piece_id=VV5WR5ZQ2ABBLBL4XRKE7T3LZN3BPWUGJCV4MVAAGUGU6ORAFLFA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: storage node overloaded, request limit: 2000 Dec 18 16:11:06.910606677 (312h42m6.781689722s) [upload=274190]: Next returned nil segment Dec 18 16:11:06.910621466 (312h42m6.781704511s) [upload=274190]: Waiting for error group managing segments... Dec 18 16:11:07.196606158 (312h42m7.067689203s) [noise=true , address=145.79.227.3:4149 , piece_id=45CAUX5P7BRQCYJWKFLN7PI5WREVIRXHRMLLKVSWENFYG6UEW4EQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.197771416 (312h42m7.068854461s) [noise=true , address=145.79.227.41:4086 , piece_id=UICMIQ2NKIFEJ2YCWWOF5F6PWQ2HY5TZOGPVYGG6OK2JFLF7TXQQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199147727 (312h42m7.070230772s) [noise=true , address=145.79.227.36:4518 , piece_id=2RHTOIK5CB5ALHDQ4RQFBMGF6TNH54BGVASFUNXDWHC4LKT5K2KA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.199886674 (312h42m7.070969719s) [noise=true , address=145.79.227.45:4231 , piece_id=JOX4PZFNYXLFFDVUOAAYTGIOVIVJHUUUIYEAWGM7B5QUUD3HRQHA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.204748963 (312h42m7.075831998s) [noise=true , address=145.79.227.48:4303 , piece_id=TXMZVMFZYBN547SMJQJKNTJFJLID4KQ6TA5USDLH6NAKBILSUBUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.211093014 (312h42m7.082176049s) [noise=true , address=145.79.227.9:4366 , piece_id=TRCJDUQFM2DGW6TY7EKGL5CMZE7H3BAAG6VTPMIF37WMES5OZGGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.222671160 (312h42m7.093754205s) [noise=true , address=145.79.227.20:4437 , piece_id=MGHOEXRWS4CLO2PPGW4GIAKWTPZ2UM4Z4OB7CWB655BPKTPYFDLA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.227728892 (312h42m7.098811937s) [noise=true , address=145.79.227.35:4482 , piece_id=XTMEBLW4XONGP2EHSZ7CLOW5W4AE7FPNRL6QSC3UJHJOG2RGSUDQ, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.228294598 (312h42m7.099377644s) [noise=true , address=145.79.227.43:4159 , piece_id=QPVWRUCGIQXOY4ONIPJJIKMIYBTL6X2RDNZKA4TLBGQFE663UQGA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.274781509 (312h42m7.145864544s) [noise=true , address=145.79.227.42:4122 , piece_id=7V5NUKTDBLKY5TSU4QCL7JMOI3MXWJI57EF6BPRKPHS2IVKSNI6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.282847367 (312h42m7.153930413s) [noise=true , address=145.79.227.37:4566 , piece_id=DFT3M7SNGMSJZCJGHGDSHGYMCEQBVHEAH7LGYQAAQULE2SZ2V3YA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.447712865 (312h42m7.31879591s) [noise=true , address=45.91.116.7:4012 , piece_id=P3QUB27SEZULH7FW6A3N4QQCISDYHH2I4SQU7GF2QL7UKPCENCUA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815760922 (312h42m7.686843967s) [noise=true , address=45.91.116.11:4117 , piece_id=WVU3NPL6UD2WNE6PVPTG5QOEG23BOXANPYNWYCF6YHQTZAGMVK6A, seg_pos={0 0}, upload=274190]: Done uploading piece. err: <nil> Dec 18 16:11:07.815788705 (312h42m7.68687175s) [seg_pos={0 0}, upload=274190]: Segment reached optimal threshold of 13 pieces. Dec 18 16:11:07.816001600 (312h42m7.687084645s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43302->45.91.116.8:4047: use of closed network connection Dec 18 16:11:07.816008693 (312h42m7.687091739s) [noise=true , address=45.91.116.8:4047 , piece_id=VITDFMIM455GJWVASUFOCMJXUN5K6EZCO7NFESGUVWP6AB6EJBSA, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816127068 (312h42m7.687210114s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37232->77.90.31.11:4555: use of closed network connection Dec 18 16:11:07.816169560 (312h42m7.687252625s) [noise=true , address=77.90.31.11:4555 , piece_id=J5MHSUO6TI3LRM4P75AXK5SKBLB5YU3IJUJ3KKUVAM3CMLC2M66Q, seg_pos={0 0}, upload=274190]: Long tail cancellation due to optimal threshold Dec 18 16:11:07.816209366 (312h42m7.687292411s) [seg_pos={0 0}, upload=274190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 16:11:07.816354492 (312h42m7.687437547s) [upload=274190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 16:11:07.816362497 (312h42m7.687445542s) [upload=274190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 16:11:08.111101419 (312h42m7.982184454s) [upload=274190]: Done uploading segments... Dec 18 16:11:08.111140293 (312h42m7.982223338s) [upload=274190]: Upload finished. err: <nil> Duration: 30.003187359s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X.X...X.....X..X........X...................................] duration: 3.440872837s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:11:30 UTC) orbiter warning: context deadline exceeded Duration: 3.440905981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 29.558643059s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:11:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 29.558669879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 8.546991245s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 16:11:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 8.547011073s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [XX.X...X.....X..X........X..................................] duration: 3.628506336s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:10:47 UTC) orbiter warning: context deadline exceeded Duration: 3.628533737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.982391504s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 16:10:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 6.982423536s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.XX.X...X.....X..X........X.................................] duration: 3.837189513s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:10:00 UTC) orbiter warning: context deadline exceeded Duration: 3.837204473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.216575368s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:09:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 7.216606589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 8.40196868s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 16:09:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 8.401996143s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X.XX.X...X.....X..X........X................................] duration: 3.572357569s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:09:40 UTC) orbiter warning: context deadline exceeded Duration: 3.572375934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 27.11274177s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:09:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 27.112768321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X.XX.X...X.....X..X........X...............................] duration: 3.482160716s trace: https://jaeger.dev.storj.io/trace/3bbed8da18007f9d FAIL (2025-12-18 16:09:18 UTC) orbiter warning: context deadline exceeded Duration: 3.482187245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 14.630312331s trace: https://jaeger.dev.storj.io/trace/731e54930b059595 FAIL (2025-12-18 16:08:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== H3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.771804627 (312h23m42.642887662s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772238212 (312h23m42.643321247s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.772745208 (312h23m42.643828253s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.773443036 (312h23m42.644526082s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774276884 (312h23m42.645359929s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.774820620 (312h23m42.645903665s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775303640 (312h23m42.646386685s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.775819863 (312h23m42.646902908s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.776341927 (312h23m42.647424972s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.777617395 (312h23m42.64870044s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.780616997 (312h23m42.651700032s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.782153803 (312h23m42.653236838s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.783722671 (312h23m42.654805706s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.784905293 (312h23m42.655988328s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.789094610 (312h23m42.660177655s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:42.793905762 (312h23m42.664988797s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Uploading piece... Dec 18 15:52:44.156786363 (312h23m44.027869398s) [upload=273925]: Next returned nil segment Dec 18 15:52:44.156802614 (312h23m44.027885659s) [upload=273925]: Waiting for error group managing segments... Dec 18 15:52:44.480271684 (312h23m44.351354719s) [noise=true , address=64.49.13.140:4507 , piece_id=7BCG2CTY2ANVWQZW3EYL4TS35R2A45HKNRLZ67ELL4FJTH542FHA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:45.664622889 (312h23m45.535705934s) [noise=true , address=145.79.227.44:4217 , piece_id=KHDEVZWDCIPMPLD3EUMXVKWTYAJHFDKJV3UI4YN3LJKVUA75GWBQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:46.287493630 (312h23m46.158576675s) [noise=true , address=77.90.31.13:4658 , piece_id=VLB7ZWDWZZWONDW5FZKMUSIGRFV7R5HCKSWLPMBRDSRTMI2HFCQQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.122165827 (312h23m47.993248872s) [noise=true , address=77.90.31.15:4708 , piece_id=UVKZ3AC2RJ5MWTEYUENTWFAWLJ2VQEAEVBZOKDTSCSJYM7XWNJNA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.428773722 (312h23m48.299856757s) [noise=true , address=45.91.116.12:4254 , piece_id=DW55EAX67QSNPSIS4BGKQAY22KYO4GBFRCTZXWWPSS4P7YMGD5AQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.927584539 (312h23m48.798667584s) [noise=true , address=77.90.31.19:4869 , piece_id=RAJ5F5Q2JSPKAS3A6YENOQRROYGW2V7ULOAOFWYCXVTNMORXQG6A, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:48.976518329 (312h23m48.847601364s) [noise=true , address=77.90.31.14:4681 , piece_id=ZBZOYFGEHX3RP7RWOY7JCPY3C2QUTQQGO2HTZ6L4QVH5EGMI3DSQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:55.942553429 (312h23m55.813636474s) [noise=true , address=45.91.116.14:4207 , piece_id=XRQ546COPCGGWDEUCZSZUF57UN4UZHJYUYI7O23YCJJIAFJSOQOQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:52:59.500618174 (312h23m59.371701209s) [noise=true , address=64.49.13.169:4706 , piece_id=KWMC7MJGB5QIHC7CWB24QQMC43TAN4I4MCCICYWEDFJ2T6CQYH7Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:06.299937143 (312h24m6.171020188s) [noise=true , address=5.231.73.2:4020 , piece_id=XHWIEAGGCXHN33YNLGRNXW6PPK2C2V7PGJIJN3H552ALUGZPJ5XA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:11.602907282 (312h24m11.473990328s) [noise=true , address=77.90.31.12:4623 , piece_id=TQNNH3DGXXXKRAWTV65KGKGGEREMTTAYQONYMTZ26F4DDPXORDDA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:12.268927408 (312h24m12.140010443s) [noise=true , address=45.91.116.5:4305 , piece_id=POSRUBE7QBGXKXQFB2OYFNYCUAES7KA4RRK4TG65PXHM6MJB3MUA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.690663477 (312h24m21.561746532s) [noise=true , address=5.231.73.5:4025 , piece_id=3BJ27XBWSXISSX4S424JWWMXEJ5CG6AI2EQWILEMJWLL74OCKKPQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818108708 (312h24m21.689191743s) [noise=true , address=77.90.31.11:4559 , piece_id=CYYQNWTKEHCNW6OEGJK4IAGJPRPOZW5F43I4M5E5BGU2E65KC4NQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: <nil> Dec 18 15:53:21.818129448 (312h24m21.689212483s) [seg_pos={0 0}, upload=273925]: Segment reached optimal threshold of 14 pieces. Dec 18 15:53:21.818409121 (312h24m21.689492166s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vPKcqLhopcpFxZ1LXwD8JXPLJB8a6HKfGnoyXsvKaL2oWUNao): protocol: read tcp 188.166.225.170:38560->45.91.116.6:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 18 15:53:21.818417887 (312h24m21.689500922s) [noise=true , address=45.91.116.6:4987 , piece_id=7KWVFQJDL623PVRLHYORTG3IPO3P2DIPRG2KRLM62O24HOEAE4PQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818434829 (312h24m21.689517874s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818448335 (312h24m21.6895314s) [noise=true , address=77.90.31.18:4838 , piece_id=ZCXKDM7QUNNDC3YITI7IRSXSMV22VRE4WX2N6J272Q7PQP577X6Q, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818589043 (312h24m21.689672078s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 18 15:53:21.818595295 (312h24m21.68967833s) [noise=true , address=77.90.31.16:4741 , piece_id=FRPEKDSBVLL3RP6EEM7HAHFRGC2O6DFNRED4GCBG377QJFRAWPNQ, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818805094 (312h24m21.689888139s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa): protocol: context canceled; context canceled Dec 18 15:53:21.818816436 (312h24m21.689899481s) [noise=true , address=77.90.31.21:4892 , piece_id=TRS6TT7X2GSZ5DHNFOQTO4JTPDRQA7IQO6MEZK74NBM6KOTR5RFA, seg_pos={0 0}, upload=273925]: Long tail cancellation due to optimal threshold Dec 18 15:53:21.818834941 (312h24m21.689917986s) [seg_pos={0 0}, upload=273925]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 18 15:53:21.818943979 (312h24m21.690027014s) [upload=273925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:53:21.818952254 (312h24m21.690035299s) [upload=273925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:53:22.122002984 (312h24m21.99308603s) [upload=273925]: Done uploading segments... Dec 18 15:53:22.122026610 (312h24m21.993109655s) [upload=273925]: Upload finished. err: <nil> Duration: 14.630340674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 9.450306931s trace: https://jaeger.dev.storj.io/trace/4dc423e4ef602829 FAIL (2025-12-18 16:08:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ruq1atBi7bkJYK7Kqv63js28sW75imA1SwTytzLneZqRRSGFU, Piece ID: 3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q) context deadline exceeded; piecestore: (Node ID: 12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, Piece ID: VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ) context deadline exceeded; piecestore: (Node ID: 12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, Piece ID: ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ) context deadline exceeded; piecestore: (Node ID: 12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, Piece ID: D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA) context deadline exceeded ======LOGS====== , upload=273916]: Uploading piece... Dec 18 15:52:16.800294444 (312h23m16.671377489s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.800761793 (312h23m16.671844838s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.801230506 (312h23m16.672313541s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802046570 (312h23m16.673129605s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802518719 (312h23m16.673601764s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.802948096 (312h23m16.674031142s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803522590 (312h23m16.674605635s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.803928834 (312h23m16.675011889s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804343403 (312h23m16.675426448s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:16.804770406 (312h23m16.675853451s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Uploading piece... Dec 18 15:52:17.618176256 (312h23m17.489259291s) [upload=273916]: Next returned nil segment Dec 18 15:52:17.618190322 (312h23m17.489273357s) [upload=273916]: Waiting for error group managing segments... Dec 18 15:52:18.010105758 (312h23m17.881188803s) [noise=true , address=185.167.204.206:4044 , piece_id=Q7BGQP3MGBM6VNONJT2HX5MCHN7WUOLGHIWOF3RFZQTJMVGGBIVQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010149151 (312h23m17.881232186s) [noise=true , address=185.167.204.206:4064 , piece_id=WNJZJDSVJPTR5KQ64PYQ4SEPFZK54D75V4GVXQDDV6GJAYUGPSJA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.010763120 (312h23m17.881846165s) [noise=true , address=185.167.204.206:4041 , piece_id=LOKLN2NG4VKKKIW2IXR7THOJFPCK4RYJOH5YUE2QNWBDAK2QRU7A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.017266895 (312h23m17.88834993s) [noise=true , address=185.167.204.205:4022 , piece_id=LB73J4CHJJXRZJSGXKX436LOFAW3HFNIBEYCT74NGI7RLP7V5PYA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.018585205 (312h23m17.88966824s) [noise=true , address=185.167.204.205:4020 , piece_id=76OPDAFRCXPV5LKIRTL5NJRAF7DO47TRAW55DNNNFCU3BZP46CGA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023654298 (312h23m17.894737343s) [noise=true , address=185.167.204.206:4046 , piece_id=K63LIE2OJISTLK5VYWG7UDGLISE625UAEKMJDYYBY7Z3VMG4P6FA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.023722077 (312h23m17.894805122s) [noise=true , address=185.167.204.206:4054 , piece_id=NXKHMK32XH2KRLQDR5OPZJ73TDF6EGT6KAI2CETA4D736KOQEAGQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.024174068 (312h23m17.895257103s) [noise=true , address=185.167.204.205:4016 , piece_id=UEPVXDBTIKPWEVUOJUPDDQHOC5DNTZMZHPITJUHW5G3CMT4FS66Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.685043376 (312h23m18.556126421s) [noise=true , address=81.171.14.70:4032 , piece_id=AUPHXZZVFQ7WUDYHO7SJETO2Q3T4GOH6SGIGGEN5O4FAAEHPY7CA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.687141842 (312h23m18.558224887s) [noise=true , address=81.171.14.70:4031 , piece_id=VVE7AA3BZDJNACAGNYRAEENX7SM5PFXAZIMGP5U4DFDXKACEV5GQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.701709071 (312h23m18.572792116s) [noise=true , address=81.171.14.70:4026 , piece_id=NAWJ3H3UR7736TDO7PGATRY6UUD2YOOWMBPBXHWKZJ4CZA33YU7Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:18.728991982 (312h23m18.600075028s) [noise=true , address=81.171.14.70:4035 , piece_id=6ZK4TNNQXZI7SCTUQEE3IR7Y3IAWYRZCYKUIP4QY62IK2YIGQKLA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.092848298 (312h23m18.963931333s) [noise=true , address=81.171.14.70:4025 , piece_id=3SHBWSFUEZP3MYLN3RFVI4RAMQT2FQGE3OAC5HRUAAS2RUEMII4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.159575150 (312h23m19.030658185s) [noise=true , address=81.171.14.70:4043 , piece_id=4PG52IAICFWKISMZZBYNMGLQ4T53ZS2YMIVQO4GE3IBWSZEF4EKA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.176613174 (312h23m19.047696219s) [noise=true , address=81.171.14.70:4034 , piece_id=D3EUQ6NTTTLK6BK5M2BOBNJYX5LMVZP4BR7HLUHH33WTWPWJ7QRA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.222812387 (312h23m19.093895422s) [noise=true , address=81.171.14.70:4041 , piece_id=FLIJBFF3RXYC7QK2GPQGHYTGJKMSOX3KWO6J3D5VBM75UEWHA3DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.330524051 (312h23m19.201607086s) [noise=true , address=89.149.201.147:4009 , piece_id=XDTDNJYDATCXXEYJTYZFAQXRAVDCE6PNTYL7S3GLROSIOPIRMATQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.397139080 (312h23m19.268222125s) [noise=true , address=81.171.14.70:4030 , piece_id=YVVTHRU6JH566X34XRAZZQHTLUXP6GFGSMJEAZUKVBEPE2M5ZCHA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.445151815 (312h23m19.31623486s) [noise=true , address=81.171.14.70:4039 , piece_id=43OK6J4NWJDL6U3XCUDV3EU4XMTET7BQJDVB6O6TS6FBVWSUKV6A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.486605222 (312h23m19.357688257s) [noise=true , address=81.171.14.70:4042 , piece_id=ZANXBNFOKP43CXSBWUPCSKFD4YKWAXF3E2JC2VQEECWRUBVETPTQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:19.956600187 (312h23m19.827683252s) [noise=true , address=89.149.201.147:4018 , piece_id=G7E2TBVZYKZLXAXBYQ7W2EDVUWZ4IEKR6S5TVDA7MXEVPIKNBIZQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.200853234 (312h23m20.07193627s) [noise=true , address=81.171.14.70:4029 , piece_id=N5V53D7LRBJ6PMVYDVMIG5MTJR7JXP3ZDVQOEZXCISIUWKIYIEWQ, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.327395857 (312h23m20.198478902s) [noise=true , address=89.149.201.147:4008 , piece_id=P7S5VCFRPVQSDZ2B3ZTZAGPUWTYQTJKFIMAMD7WOD5OHYAGRS5GA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.404743533 (312h23m20.275826598s) [noise=true , address=89.149.201.147:4023 , piece_id=CQKV6QJPULAIWCJMUAUKO6NDDQYU23I46FR5WJBGAZAFZQQZXA2A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.596469639 (312h23m20.467552674s) [noise=true , address=89.149.201.147:4003 , piece_id=QF2MD2RKS7SVJCSWQVHAVTFIC2KUQ327ORD5GQSG3HW444V6OI4Q, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731704011 (312h23m20.602787056s) [noise=true , address=89.149.201.147:4022 , piece_id=XQVWF4NBDRBFEX2NQ6QJ4BAFX7FZKWBZ6VTLW3O3SR5OL6M3T43A, seg_pos={0 0}, upload=273916]: Done uploading piece. err: <nil> Dec 18 15:52:20.731728959 (312h23m20.602812004s) [seg_pos={0 0}, upload=273916]: Segment reached optimal threshold of 26 pieces. Dec 18 15:52:20.731944810 (312h23m20.603027855s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HSPWfW4L5gN2YoEV8rcqhQtVFv83M21nseT9MZSTgwJo76yGx): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731954538 (312h23m20.603037573s) [noise=true , address=81.171.14.70:4037 , piece_id=SMUBVADFQFVM4COQAVKRWYVPABONLGEO7TCNZTOZTZQT4TSIGNAA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.731986038 (312h23m20.603069083s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MUVGecysnTSFcJoHUL7QMWb3h6dtXgUd7FbA672fQCyEWhePr): protocol: expected piece hash; context canceled; EOF Dec 18 15:52:20.731997591 (312h23m20.603080646s) [noise=true , address=81.171.14.70:4044 , piece_id=7E6AE27KLFEVVXNNQLU4L4RALPLJOSRNM6MCHIQKSRAH5JCVV7DA, seg_pos={0 0}, upload=273916]: Long tail cancellation due to optimal threshold Dec 18 15:52:20.732021075 (312h23m20.60310412s) [seg_pos={0 0}, upload=273916]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 18 15:52:20.732168066 (312h23m20.603251111s) [upload=273916]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 15:52:20.732175329 (312h23m20.603258374s) [upload=273916]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 15:52:20.962827968 (312h23m20.833911003s) [upload=273916]: Done uploading segments... Dec 18 15:52:20.962851803 (312h23m20.833934849s) [upload=273916]: Upload finished. err: <nil> Duration: 9.450340947s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)